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Foreword 



It can be argued that the word “partnership” should be used cautiously and with care. After all, it does have a specific legal meaning 
about a shared, contractual business venture. The theme of this io8th Annual Meeting program was chosen with a much broader 
appreciation for the term: through partnerships “partners" share a common activity or interest. In fact, this Collection of Papers is 
testimony that both types of partnerships are growing in importance to our membership. 

We claimed in the 1990s that higher education was undergoing a major transformation. Technology fueled much of the change. So also 
did the changing demographics of our students. Today finances have come to be another very significant driver for change. Effective 
learning environments demand more and more use of technology; services to diverse students require a richer array of programs and 
support services; at the same time funds from states dwindle and investment portfolios fail to perform. If higher education is going to 
meet growing demands, it inevitably must turn to new and different organizations to help share the design, support, and costs. New 
business partnerships, some among colleges and universities but many with private corporations, come into play. These are, in fact, 
partnerships in the legal, contractual understanding of the word. We need to know more about these, to identify what the appropriate 
quality assurance interests might be in them, and to disseminate best practices in them. Several of the papers presented in this 
Collection of Papers give us significant help in all these matters. 

We have an even larger growth of partnerships that emerge from sharing of common interests. How do colleges and universities work 
together and with the K-12 sector to create effective educational pathways for students? How do groups of institutions create ways to 
share the endeavors of creating effective e-Learning courses and programs? What issues must be solved for two or more colleges to 
create a shared degree or a shared set of student services? These types of partnerships figure prominently in many papers in these 
volumes. 

Accreditation is really a tremendous exercise in creating partnerships as well. Within any given college, the success of a self-study 
process engages the Commission, the self-study team, and all the constituents of that college. The Commission’s peer review processes 
depend on the willingness of its members to share the talents ofsite visitors and decision-makers. The Commission strives to build and 
maintain good relationships with state and federal governments also concerned about the quality of colleges and universities. Several 
of the papers in these volumes speak to these partnerships. 

A considerable number of papers focus on two major Commission projects involving partnering with affiliated institutions. AQIP consti- 
tutes the most path-breaking experiment in recasting accrediting relationships, relationships of institutions to the Commission and 
institutions to each other. From these essays we learn about the power of this new approach. Effective and meaningful assessment of 
student learning still stands as an unmet goal of many colleges. These essays are the richest collection to date of accounts of the 
successful efforts of some institutions to create a new culture supportive of assessment. 

Last, but not least, with the adoption this February of the new Criteria for Accreditation, the Board of Trustees not only integrated 
institutional and business partnerships into accreditation standards but also challenged its affiliated organizations to focus on the 
future, on their support for learning, on their connectedness, and on their attention to their own distinctivenesses. Not many papers in 
this volume speak to directly to this new partnership meant to “serve the common good byassuringand advancing the quality of higher 
learning." But discussion about it will be central to the Annual Meeting itself and the fruits of the new partnership inevitably will fill 
pages of future editions of the Collection of Papers. 



Steven D. Crow 
Executive Director 



March 1, 2003 
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Preface 



On behalf of the Commission, I am pleased to present the 2003 edition of the Collection of Papers on Self-Study and Institutional 
improvement. Nowin its nineteenth year, the Collection of Papers has moved beyond a supplement to the Meeting presentations to 
be a remarkable resource throughout the year for all who are interested in issues of higher education quality. We are grateful to our 
speakersfortheirgenerous contributions to the work ofthe Commission through these papers as well as through their presentations 
at the Annual Meeting. 

With this edition, the Collection of Papers moves into a new phase of publication. The contributions of our speakers have grown in both 
number and substance. Topics once addressed in two or three papers, are now featured in twenty or more. It is no longer practical or 
useful to publish these papers in a single volume. Therefore, this Collection is presented as follows: 

Volume 1. Establishing and Sustaining Effective Connections flows from the theme of this year’s meeting, providing a 
wide variety of examples of new partnerships being forged by higher learning organizations and highlighting some state, 
regional, and national initiatives. 

Volume 2. Organizational Effectiveness and Future Directions focuses on the relationships among mission, planning, 
and organizational change; quality improvement in higher education, including a number of papers from institutions 
participating in the Commission’s Academic Quality Improvement Project; the various challenges and opportunities 
offered by technology; and the role of effective learning environments in achieving institutional goals. 

Volume 3. Promoting Student Learning and Effective Teaching features three chapters on the role of assessment in the 
improvement of student learning; a fourth chapter is devoted to assessing and reforming general education; the fifth 
chapter focuses on assessing and supporting effective teaching. For the fourteen years ofthe Commission’s Assessment 
Initiative, the Collection of Papers has highlighted institutional efforts to assess student academic achievement. It is impor- 
tant to note that discussions of assessment efforts are not limited to this volume, but appear in numerous papers throughout 
the four volumes. 

Volume 4. The Self-Study Process for Commission Evaluation, offers useful advice on organizing and conducting self- 
study and undergoing a team visit based on actual experience; it includes one case study on seeking and attaining 
initial status with the Commission. 

Producing a book of this size in five weeks requires significant team effort. Special thanks are given to the following individuals who 
made the 2003 Collection possible: Larissa Kessler, for her help in processing initial submissions and preparing files; Sybil Sosin,for 
her valuable editorial assistance; Gerald Van Kollenburg, for his extraordinary assistance with the layout, particularly the charts and 
graphics; Kathleen Herring, forthe beautiful cover designs; and Aaron Marsh of Honi Graphics, for always getting the book printed in 
time for the Meeting. 

The Commission invites yourcomments about the Collection of Papers and welcomes your suggestions forfuture topics forthe Annual 
Meeting program. I hope thatyou will consider participation as a speaker at a future Meeting. The strength of the Annual Meeting lies 
in the willingness of our institutions to share their experiences with others. I look forward to seeing you in at the Meeting. 



Susan E. Van Kollenburg 
Editor 

Associate Director for Programs, Publications, 
and Member Services 



March 1, 2003 



Ed. note: The name of the Commission was changed from the Commission on Institutions of Higher Education to The Higher 
Learning Commission effective lanuary 1, 200ir In their papers, authors may have referred to this organization as the North Central 
Association, theCommission on Institutions ofHigher Education, orThe Higher LearningCommission. Information aboutthe name 
change is available on the Commission’s web site: www.ncahigherlearnlngcommission.org. 
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Developing the Scholarship of Assessment: 
Guidelines and Pathways 

Thomas Anthony Angelo 



Focus and Audience 

This paper focuses mainly on small-scale, faculty-directed assessment efforts and mostly on how and who questions. Specifically, It 
suggests how to build in success by learning from prior reform efforts and by taking a scholarly approach to assessment. It considers 
who itisthata “scholarship ofassessment”(SoA)mightaim to inform and influence and, consequently, whoamongthe faculty should 
be encouraged to practicetheSoA.lt also suggests how bestto engage, support, and sustain faculty practitioners in high-quality, high- 
impact SoA efforts. While it is intended primarily to help faculty leaders and academic administrators responsible for promoting and 
leading SoA efforts, it should also be useful to those advancing other academic Innovations. 



Underlying Premises 

Four premises undergird the arguments and guidelines that follow. The first is that the SoA can improve educational quality only if it 
engages enough of the right “producers” and “consumers.” That is, a significant fraction of influential faculty leaders (scholarship 
producers) must engage and succeed in the SoA. At the same time, a much larger proportion ofthe faculty (scholarship consumers) 
must be convinced to take these activities seriously as scholarship and to apply SoA results to their teaching. The second premise is 
that productive, influential faculty will engage in the SoAonly if they find it intellectually compelling, professionally rewarding, and 
relatively unburdensome. The third underlying premise is that-to be intellectually compelling, rewarded, and effective-SoA efforts 
must be designed and implemented according to the highest, most broadly accepted standards of academic scholarship. 



Sources 

The necessary conditions and guidelines for success are drawn from three main sources: the growing literature on the scholarship of 
teaching and learning, research and best practice studies in higher education assessment and faculty development, and literature 
on the diffusion of innovations and organizational change. 



The SoA’s “Parents”: Assessment and the Scholarship of Teaching 

Assessment and the scholarship of teachingare broad-based academic reform movements. The scholarship of assessment is a hybrid 
of these two movements. Given that the promise of SoA derives largely from its dual parentage, it seems worthwhile to consider the 
strengths and weaknesses of both. 

□ Assessment 

According to Palomba and Banta, “Assessment is the systematic collection, review, and use of information about educational 
programs undertaken for the purpose of improving student learning and development” (1999, p. 4). After fifteen years of a 
widespread assessment “movement,” there is now broad agreement among accrediting agencies, disciplinary and professional 
associations, administrators, and faculty opinion leaders that improving student learning is (or should be) the primary goal of 
assessment. That all of our accrediting agencies require assessment and that virtually all American colleges and universities 
claim to be practicing it (El-Khawas, 1995) are evidence of its wide and still growing influence. At this point in its development. 



Many of the ideas in this paper have been drawn directly from Angelo, T. A. “Engaging and Supporting Faculty in the Scholarship of Assessment: 
Guidelines from Research and Best Practice,” Chapter 10 in T. W. Banta et aL, Building a Scholarship ofAssessment. San Francisco: jossey-Bass, 2002. 
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assessment is a relatively mature innovation that can claim a range of resources, a growing literature, and a large network of 
practitioners (see Gardiner, Anderson, & Cambridge, 1997; Banta, Lund, Black, & Oblander, 1996). 

□ The Scholarship of Teaching and Learning 

More recently, widespread consensus has begun to emerge among these same leadership groups on the need to expand the 
range of scholarly activities and faculty roles that are encouraged, evaluated, and rewarded. This notion of an expanded model 
of scholarship, and of the scholarship of teaching in particular, was first championed by Boyer (1990) and Rice (1991). Building 
on their work, scholars associated with the Carnegie Academy for the Scholarship of Teaching and Learning (CASTL)-notably 
Giassick, Huber, and Maeroff (1997) and Hutchings and Shulman (i999)-developed definitions, examples, and standards for 
evaluating this form of scholarly activity. 

But what is the scholarship of teaching or the scholarship of teaching and learning, since the two terms are used virtually 
interchangeably? According to Hutchings and Shulman (1999), “A scholarship of teaching is not synonymous with excellent 
teaching. It requires a kind of ‘going meta,*in which faculty frame and systematically investigate questions related to student 
learning— the conditions under which it occurs, what it looks like, how to deepen it, and so forth— and do so with an eye not only 
to improving their own classroom but to advancing practice beyond it” (p. 12). Throughout, CASTL materials characterize it as 
discipline-based, public (“community property”), open to critical peer review and evaluation, and capable of being adapted and 
used by other teachers in the same discipline. With support from the Carnegie Foundation, the American Association for Higher 
Education, and the Pew Foundation, CASTL has created an infrastructure for developing and disseminating the scholarship of 
teaching and learning, along with a number of useful publications <see http://www.carnegiefoundation.org/>. As of late 2002, well 
over one hundred campuses are formally involved in these efforts, and many more are engaged in activities influenced by CASTL. 



The Limited Impact of Assessment and the Scholarship of Teaching 

Notwithstanding these impressive achievements, neither assessment nor the scholarship of teaching has yet to make the deep and 
lasting impact on teaching and learning quality or academic culture that proponents have hoped for. In their review of the last two 
decades of reform attempts, Lazerson, Wagener, and Shumanis assert that “efforts to improve teaching and learning have been 
supported only in part by faculty and institutions as a whole, with results that are neither significant nor pervasive” (2000, p. 14) and 
conclude that “a genuine teaching-learning revolution seems far away” (p. 19). The same can be said of the scholarship of teaching 
at the end of its first decade. If the scholarship of assessment is to succeed, we must find ways to get unstuck and penetrate deeper 
into academic culture. That requires first correctly diagnosing the reasons that promising and widely supported academic change 
efforts, such as assessment, so often fail to meet expectations. 

In a 1997 /W/yf Bulletin article, Peter Ewell offers a compelling diagnosis of failed academic reforms, noting that they have “been 
implemented without a deep understanding of what ‘collegiate learning’ really means and the specific circumstances and strategies 
that are likely to promote it” and “for the most part been attempted piecemeal within and across institutions ( p. 3). I would add a third, 
related reason: Reforms have been implemented without a deep understandingof how faculty themselves learn and develop, of how 
change occurs in academic culture, and of the most effective strategies and approaches for promoting lasting change. 



The Scholarship of Assessment 

By all rights, the scholarship of assessment should bean attractive and effective innovation, given that it has the potential to respond 
to many real, widespread needs in higher education. For example, it has long been recognized that most American faculty members 
do not, in fact, engage in the “scholarship of discovery”-in the traditional disciplinary forms of research that result in publication in 
refereed journals and grants— and that most faculty members both care about teaching and believe that it is undervalued. Partly for 
those reasons, many American universities are now revising or have already revised their retention, tenure, and promotion policies 
to include a broader conception of scholarship and to reward a broader range of scholarly activities. A likely expectation of 
administrators and trustees backing these changes is that a greater proportion of the faculty will engage in documentable and 
meaningful scholarly activities. Thus, while this broadening of options will benefit faculty members who are already engaged in less 
traditional forms of scholarly activity, it may also impel significant numbers to develop new skills and interests. 

To respond to changed expectations and take advantage of these wider options, many faculty members will need training and support 
in systematic, straightforward ways to do scholarly work on teaching and learning issues. The SoA can provide such an approach. 
Academic administrators, in turn, need more valid and useful information on teaching and learning effectiveness for personnel 
decisions, public relations, program review, and accreditation. But few institutions can afford to invest the additional staff and 
financial resources needed to generate this information through existing institutional research and assessment processes. Faculty 
engaged in the SoA could help provide such information, along with the knowledge and judgment needed to make use of it. Those 
responsible for assessment, faculty development, and accreditation need effective ways to engage and sustain faculty involvement 
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in those efforts. By engaging large numbers of faculty in applied inquiry, the SoA could respond to these organizational development 
needs as well. Consequently, the SoA holds great promise for engaging faculty in activities to document and improve teaching 
effectiveness and student learning quality that are both institutionally and individually valuable. 

But promising ideas alone-even ones that meet real needs-are not sufficient to change academic culture, as the past half century 
of attempts to disseminate innovations amply demonstrates. A short list of promising but largely unrealized reforms might include 
educational television, programmed learning, mastery learning, writing across the curriculum, computer-assisted learning, and 
multi-media instruction. 

Howcan the SoAavoid this common fate? First and foremost, realizingthe promise ofthe SoA will require its “champions” to recognize 
and apply lessons learned from previous academic innovations, both successful and unsuccessful, and from research on the diffusion 
ofinnovations more generally. Second, it will require alignment amongthree key elements: institutional systems, faculty culture, and 
leadership for change. In other words, it will require a more systematic, strategic, and scholarly approach to innovation. Takingthese 
hard-won lessons seriously can better the odds that faculty wilt engage and persist in the scholarship of assessment, and thus 
increase our collective understanding of and capacity to improve student learning. 



Some Necessary Conditions for Academic Reform 

Several interrelated conditions seem necessary, albeit not sufficient, for an academic innovation to succeed. First and foremost, the 
proposed reform has to meet strongly felt faculty needs as well as institutional priorities. Second, it must conform to the values and 
culture ofthe faculty. Third, it requires effective leadership and “followership,” including involving influential faculty members from 
the start. Fourth, it requires long-term planning and a long-term institutional commitrnent. Fifth, success requires well-aligned 
support from the key systems that affect faculty life, including tenure and promotion, merit, course and faculty evaluation, workload 
distribution, and faculty development. Sixth, it requires resources, which often means redistributing time, energies, and effort. And 
last, it requires “scaffolding,”-programs and services designed to support faculty through the difficult process of reflection, 
unlearning, experimentation, and critique that change involves. 



Whom to Involve When 

Perhaps the greatest challenge both the assessment and scholarship of teaching movements face is engaging and sustaining broad 
and deep faculty involvement. Indeed, this is a problem common to all academic innovations— and innovations generally. In Diffusion 
of Innovations, Rogers provides a framework for analyzing this problem. He sorts members of organizations into the following 
categories by their responsiveness to innovation: innovators, early adopters, early majority, late majority, and laggards (1995, pp. 
252-280). Rogers characterizes innovators as venturesome, risk- and failure-tolerant cosmopolitans who are well connected to wide 
networks of innovators outside their organizations. Though innovators are the bringers of change, they are not necessarily highly 
respected local opinion leaders (pp. 263-264). Early adopters, on the other hand, are typically well-respected local opinion leaders 
whose championing ofinnovations is necessary, if not sufficient, to make change (p. 264). These two categories typically make up less 
than a fifth of the total population of any organization. Early adopters convince the early majority, the next third or so, to try the 
innovation. 

Theassessment movement has struggled to expand and sustain faculty involvement beyond the innovatorsandearlyadopters. Unless 
newapproaches to scholarship and assessment reach beyond these first two categories of disseminators, neither movement is likely 
to become an integral part of academic culture. By extension, the same caveats hold for the scholarship of assessment. 

Ten Guidelines for Promoting, Supporting, and Sustaining the Scholarship of Assessment— and 
Other Academic Innovations— to Improve Learning 

1. Plan for a long-term campaign. Successful innovations typically take years to become part ofthe majority’s standard practice, 
often a minimum of three to five years. Consequently, it’s critical to plan for a several-stage, multiyear campaign that can 
persevere until assessment is fully integrated into the local academic culture. 

2. Engage and involve opinion leaders from the start. In general, who we begin with determines whom we end up with. Start by 
recruiting academic opinion leaders from groups we hope to involve, rather than recruiting the easy and obvious participants. 
An initiative that starts with post-graduate teaching fellows, new faculty, or the “usual zealots,” for example, is likely to be 
perceived as being useful only to those groups. Department chairs are among the most critical actors to involve. 
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3. Keep the focus on what matters most: improving student learning. The more reasons faculty members find to engage in 
assessment, the better. The main attraction and promise of assessment are its focus on improving student learning, a focus most 
faculty share. Taking this aim seriously, however, means finding valid and accessible ways to measure student learning, rather 
than just involvement or satisfaction. 

4. Identify likely costs and benefits— intrinsic and extrinsic; then lower costs, raise benefits, and look for multiplier effects 
whenever possible. Potential assessors need to know what’s in it for them, and what it will cost them. Costs of engaging in 
assessment centeraround time, effort, and frustration. While costs can never be totally eliminated, they can be minimized. Likely 
benefits are multiple and relatively easy to capitalize on. Administrators need cost-effective ways to get information on student 
learning. Faculty need meaningful research projects for professional development and career advancement. Post-graduate and 
undergraduate students perpetually need funded employment and research opportunities. Providing support for student 
apprentices on assessment projects can generate multiplier effects all around. 

5. Start with the familiar and make connections. This well-supported guideline for effective learning applies to faculty as well as 
students. In the caseoffacultylearningto use assessment, familiar elementsare the everydayteachingand testing practices they 
have observed and experienced. For faculty, making connections between their prior knowledge and experience and assess- 
ment-understanding key similarities and differences— will be critical to success. Walvoord and Anderson (1998) offer a powerful 
way to build on the familiar by engaging faculty in turning grading into a rigorous, useful form of assessment. 

6. Provide scaffolding for novice and intermediate practitioners. Most academic innovations begin with training and support for 
novices, and most end there. But most participants need ongoing support well beyond the novice phase. As assessment 
practitioners become more experienced, the amount of structure, or scaffolding, they require diminishes. 

7. Developand sustain socialsupportsforpractitioners.Changingacademic practice is likechanginganyset of habitsorpatterns. 
Support groups demonstrably improve the perseverance and success of those trying to lose weight, stop smoking or drinking, 
and develop exercise routines. In the same way, assessment support groups can increase faculty perseverance and success. 

8. Don’t pay participants to do what should become part of routine practice. Academic development projects, particularly those 
with grant funding, often offer stipends to participants for engaging in the desired new behaviors. All too often, faculty stop 
engaging in those behaviors when stipends stop. New recruits are hard to find after the money runs out. If we want faculty to 
engage in assessment over the long term, payingthem initially is a risky strategy. Instead, use available funds for books, related 
travel, student support, and the like. 

9. Insist on clear criteria, evidence, and high standards for quality. For assessment to become an accepted, consequential form 
of scholarly activity, it must be evaluated by scholars against meaningful, agreed-upon criteria. Thanks to Glassick, Huber, and 
Maeroff (1997) and the Carnegie Foundation, we now have six criteria: clear goals, adequate preparation, appropriate methods, 
significant results, effective presentation, and reflective critique. The challenge for institutions and departments is to adapt 
these broad criteria to specific needs and circumstances and to different forms of the scholarship of teaching, including the 
scholarship of assessment. 

10. Share information on efforts, findings, and successes widely. To be credible and useful, assessment results must become 
community property. If assessment is a kind of public education campaign, then the “public” will require many and various 
messages before they get the point. For many faculty members, definitions matter less than examples, and numbers less than 
narratives. Planning ahead and using assessment to gather illustrative examples and stories are critical steps in engaging faculty. 
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The Convergence of Assessment and 
Scholarship of Teaching and Learning 

David Sill, Douglas Eder, Lesa Stern, Susan Thomas, and John DenHouter 



After fifteen years of assessment and a decade of scholarship of teaching and learning, the two initiatives are converging. The best 
assessment has become the best scholarship of teaching and learning, and vice versa. This paper will discuss the background and 
history oftheseinitiatives atSouthern Illinois University-Edwardsville and will present illustratingexamplesofwherethis convergence 
is occurring: the assessment scholars program, the senior assignment, and course portfolios. The convergence of assessment and 
scholarship of teaching and learning helps address two problem areas or challenges that campuses have faced in trying to develop 
strong and effective assessment programs: ownership and depth. 

The first of these problem areas, achieving faculty ownership for the process and the results, is an ongoing challenge for meaningful 

campus-level assessment of student learning. Because thefaculty is responsible forwhat goes on in the classroom and forthe curriculum, 
faculty ownership is important if assessmentis to effect improvements in student learning. To develop faculty ownership, we must ensure 
that assessment is an activity worth doing and that the results of assessment are considered worthwhile, meaningful, and valid. 

The second of the problem areas, depth, is an often unspoken question that haunts many assessment programs: “Are we measuring 
what is important, or are we making important what we can conveniently measure?” Goals of higher education that go beyond skills 
and content are exactly what distinguish higher education from being just a higher level of secondary education. Goals of higher 
education include character and intellectual development. They include beliefs, values, morals, attitudes, ethics, and citizenship on 
theone hand, and habits ofthemindandpractice,competence,andprofessionalbehaviorsontheother.Yetcharacterand intellectual 
development are difficuitto assess. We have come to know that there is a need for a deeper assessment of student learningthan what 
is available through standardized tests, capstone projects, and student surveys. At best, these measure student mastery of skills and 
content, but more often they do not even do that. Rather, they often focus on howstudents feel abouttheir mastery ofskills and content 
instead of directly monitoring actual student achievement. Furthermore, even valid measurements of skills and content are limited 
in assessing the distinguishing goals of higher education. In addition, they provide little insight into the relationship between what 
students learn and the ways in which they learn it, both in terms of pedagogy and curriculum. The challenge is for higher education 
to apply the depth and level of intellectual rigor to the assessment of student learning that we routinely demand of our work in 
disciplinary-based research. 

Higher education does not have a good track record of involving its primary resource, the faculty and its scholarship, to the study and 
improvement of higher education itself. Trudy Banta, from Indiana University-Purdue University Indianapolis, and others have begun 
to address this challenge by thinkingin terms of the scholarship of assessment (Banta etal., 2002). SlUEhas taken a second approach 
by drawing on an independent body ofwork— the scholarship of teaching and learning, as developed bytheAmerican Association for 
Higher Education’s (AAHp Teaching Initiative and the Carnegie Academy for the Scholarship of Teaching and Learning. Over the last 
decade, assessment and the scholarship of teaching and learning have developed as separate initiatives. Even within AAHE, which 
has supported both of them, the two initiatives have been segregated into two conferences, the Forum on Faculty Roles and Rewards 
in the winter and the Assessment Conference in early summer. In spite of that separation, even from the beginningthere has been an 
aspect of the scholarship of teaching that involved student learning: 

No longer can we think of teaching in the terms of the old formula: subject-matter expertise plus generic methods (howto plan 
a lecture, lead a discussion group...) equals good teaching. Effective teaching is also a matter of transforming one’s knowledge 
of a subject in ways that lead to student learning. (Edgerton et al., 1991) 

At SlUE, as more and more faculty have become involved in those separate activities, the faculty’s work has begun to converge to the 
point where the best examples of assessment on campus are clearly scholarship of teaching and learning activities and vice versa. 



History of Assessment and Scholarship of Teaching and Learning at SlUE 

SlUE’s faculty senate included outcomes assessment in its annual goals in 1985 and again in 1986, specifying that the faculty senate 
would “work with the administration in coordinating the various activities associated with outcomes assessment” and would 
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“encourage and participate in an investigation [of] various methods of outcomes assessment” (UniversityCommittee on Assessment, 
1989). In September 1986, the Illinois Board of Higher Education (IBHE), in its Report of the Committee on the Study of Undergraduate 
Education, called for Illinois colleges and universities to “develop appropriate means to assess student progress in meeting the 
objectives of general education and the development of baccalaureate-level skills” and to “assure that assessment programs 
reinforce the maintenance of academic standards” (IBHE, 1986). 

Over the following two and a half years, SlUE developed the university assessment plan, which was approved by the faculty sen ate in 
April 1989. That plan was created and implemented based on principles of faculty involvement and leadership from the beginning, 

faculty ownership ofthe process, significant administrative support, and thehousingandcreatingof major parts ofassessment in the 

disciplines. Entry and midpoint assessment began in 1989, with the senior assignment (SRA) fully developed and implemented by all 
programs in fall 1992. Each SRA was designed to enable seniors to exhibit a general education perspective while demonstrating 
proficiency in the major. In 1995, the midpoint assessment was changed to include a broad range of activities, such as student 
portfolios and the assessment scholars program. 

In November 1994, an IBHE staff report announced a goal “to create a base for change and to encourage institutions to organize and 
expand their efforts to address faculty roles and responsibilities” (IBHE, 1994)- SlUE began its faculty roles and responsibilities 
initiative in spring 1995, with its first symposium in fall 1995 and a white paper and a formal plan that were submitted to the faculty 
senate and to the IBHE. The original steering committee chose to concentrate SlUE’s efforts on peer review of teaching and the 
scholarship of teaching and learning. In the following years, the faculty senate formed a subcommittee ofthe faculty development 
council to coordinate and provide leadership for the initiative. 

In fall 2002, SlUE’s faculty roles and responsibilities committee and the provost’s office sponsored a symposium that explored the 
connections between scholarship of teaching and learning and assessment. During that symposium, we identified the following 
common characteristics in which both activities 

o Identify an issue of teaching or learning 

o Study that issue through appropriate methods 

o Apply results to teaching practice 

o Communicate the results to others 

o Engage in self-reflection and peer review 

Scholarship of teaching and learning and assessment of student learning have become fellow travelers. Hutchings and Shulman 
captured this common core when they said that the scholarship of teaching and learning “requires a kind of ‘going meta,’ in which 

faculty frame and systematically investigate questions related to student learning-the conditions under which it occurs, what it looks 

like, howto deepen it, and so forth— and do so with an eye not only to improving their own classroom but to advancing practice beyond 
it” (Hutchings and Shulman, 1999). 

It is important to note that both assessment and scholarship of teaching and learning, which are collective faculty responsibilities, 
occur at different levels or scales of organization-in individual courses, within disciplines, and across the baccalaureate curriculum' 
(and indeed across the graduate curriculum). This paper looks at examples at each level. The assessment scholars program allows 
for an institution-wide probing of student learning as it relates to the statement of objectives for general education and the 
baccalaureate degree, in this example with a special focus on student writing. The senior assignment, in psychology for example, 
allows for formal assessment of student learning at the program level in the undergraduate psychology major and general education 
by the psychology faculty. The course portfolio provides a mechanism at the course level for a rigorous, scholarly study of course 
pedagogy, student learning, and the connection between the two. In the example from art and design, faculty developed course 
notebooks for the foundation art studio courses to allow common understanding of student learning goals among faculty who teach 
those courses and accordingly to allow for a shared means of assessing that student learning across sections and instructors. 



Assessment Scholars Program 

The assessment scholars program connects the assessment of student learning outcomes and the scholarship of teaching and 
learning in a traditional scholarly format. In 2000, the program awarded fellowships to faculty for undertaking formal research on an 
announced student learning topic. Selection is based on a competitive peer review of formal research proposals. 

For the past eight years, the SlUE assessment office has collected student portfolios. Our library of student work allows us to ask 
questions such as the 2000-2001 focus topic, “To what extent and by what means do students learn to write betterat SlUE?” The three 
scholars took varied approaches. One addressed programmatic issues in comparison with writing programs at other universities; 
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another addressed the extent to which the writing quality of juniors and seniors is better than that of freshmen and sophomores; and 
the third addressed the kinds of written feedback that professors are providingto students on their papers. The assessment scholars 
examined written work and professors* responses within the portfolios as they would any primary sources and arrived at conclusions 
supported by evidence. The scholars wrote manuscripts and submitted their findings to refereed journals for peer review (note that 
confidentiality of individual findings was maintained and results were published in the aggregate). 

In general, universities delegate issues of writing to the English department, whereas most writing is done outside the English 
curriculum. One important aspect of this first round of assessment scholars was that the faculty were from philosophy, educational 
leadership, and speech communication, which provided a more diverse and fresh perspective on writing than if the faculty had come 
from an English background. Involving a broad spectrum of faculty to address a university-wide issue such as writing has been one 
of the benefits of the 2000-2001 assessment scholars program. 

The longitudinal portfolio data collected from students with diverse career goals and majors provided another important aspect of this 
program. SlUE’s portfolios are dictated portfolios in that they contain pieces of work required by the assessment office, not items 
selected by the students. The portfolios contain both drafts and final versions of papers, as well as essays and other smaller writing 
assignments that might be overlooked in a study that examined more traditional English writing samples. Yet all forms of writing can 
be valid for assessment purposes. A memo or a brief may be a more legitimate form of writing sample in the business or legal world 
than a five-paragraph academic essay. Because the dictated portfolios include all student papers, not just those that are selected by 
students or faculty as best practices, both student and faculty paper selection bias is removed, avoiding a weakness of many studies 
assessing student writing. This facet is particularly important for the assessment scholars project that focused on faculty comments 
on student papers. Also, SlUE faculty did not routinely know which students participated in the portfolio project, although the 
information was available publicly. We can be confident that the comments on the portfolio papers were typical and real and 
represented common practice. 

The convergence of assessment and the scholarship of teaching and learning became visible as the assessment scholars presented 
their thoughts, methods, and findings to the faculty as a whole. Several informal brown-bag sessions and more formal presentations 
(such as the SIUE fall faculty symposium) have highlighted assessment and the assessment scholars* work. After the presentations, 
faculty from all departments took part in lively discussions that highlighted the issues, problems, and struggles that faculty and 
students faced in improving student writing, as well as success stories and sharing strategies faculty have found particularly effective. 
Handouts were provided so that the faculty had tips and advice, based on a wealth of research, on improving student learning, 
including how to improve feedback given to students. As a result, some faculty members have now rethought the way they design, 
implement, and comment on student writing assignments. 

The assessment scholars program is just one avenue for inquiry, reflection, and debate at SIUE. It has helped enrich a tradition of thinking 
through, reflecting on, and evaluating teaching and learningovertime. Assessment becomes even more meaningful to faculty when the 
assessment ofwhatandhowstudentsarelearning(theirlearningstyles,challenges,successes,etc.)in combination with analysesofwhat 
faculty are doing (their processes, teaching styles, assignments, etc) actually help both students and faculty meet their goals. 



Senior Assignment 

At SIUE, the senior assignment (SRA) is an academic simulator, that is, a reflective and scholarly engagement between student and 
dedicated professor resultingin a visible product or behavior. Much like an airplane cockpit flight simulator, SIUE*sseniorassignment 
asks students to undertake a complex task under the supervision of an instructor such that teaching, learning, and assessing take 
place simultaneously. Under the best conditions, assessment is embedded in the teaching and learning process, and the mechanics 
of assessment are transparent to the student— again, just like in the cockpit flight simulator. It is authentic assessment in that it 
appraises student performance of a real task in a real or simulated environment under supervision of faculty. The SRA is embedded 
in learning and teaching, matches departmental goals, and is departmentally owned. It is a high-stakes assessment that is viewed 
by the faculty and includes a liberal education component. 

The SRA in SIUE*s psychology department is an ever-evolving process. Initially, students had only one option to complete the SRA. 
Students would submit a five- to seven-page paper (in APA style with appropriate references) in which psychological knowledge and 
techniques would be related to an issue of social, ethical, health, or political importance. The psychology faculty would review the 
paper. Unfortunately, it soon became apparent that the students were not taking the paper seriously and were submitting inferior 
work, making it impossible to assess whether they were achieving the departments goals for baccalaureate education. By itself, this 
is an important assessment finding. To improve student performance and the effectiveness of the SRA, the faculty added two options. 
In addition to the paper, students could 

1. design and carry out research as a requirement in an upper level course. Students must explain and defend their projects in 
formal paper reading or poster sessions (attended by other students and faculty in the department). The best posters are 
submitted to the annual meeting of the Midwestern Psychological Association for presentation; or 
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2. design their own independent research projects under faculty supervision. Results of this research are also presented atthe 
annual meeting of the Midwestern Psychological Association or at the departmental paper reading/poster sessions. 

Students took the two new options much more seriously, thus giving the department the assessment vehicle it desired. Even though 
the rigor of the paper option was also increased, it has since been dropped. 

The form of the SRA is not the only aspect of the assessment process that has evolved. In examining the SRA options, the department 
determined that the goals upon which the SRA was based were overly broad and difficult to measure. For example, an initial goal was 
that students were expected “to be aware ofthe cultural, social, historical, ethical, and economic impacts of psychological knowledge 
and techniques.” As awareness is very difficult to measure, this goal has been restated, thereby sharpening the faculty’s own 
expectations and standards. The goals now state that students will demonstrate the ability to relate research findings to a broader 
cultural, social, historical, ethical, or economic realm, and that ability is measured across students as a program goal through primary 
trait analysis. 

Just as the SRA is a scholarly endeavor, so too is its evolution. By examining the data every step ofthe way, the department learned 
to restate its baccalaureate goals in measurable terms, employ primary trait analysis, refine its SRA options, and raise its standards. 
This has permitted the psychology department to develop the SRA into an effective assessment tool and to improve student learning. 



Course Portfolio 

MHE’s teaching initiative developed course portfolios as one strategy for approachingthe peer review of teaching. Acourse portfolio 
is a scholarly project, with the course as the unit of inquiry, that begins with an important question and ends with a text that can be 
shared with peers, who can then build on the work. At the AAHE Forum on Faculty Roles and Rewards in 1996, the work ofthe teaching 
initiative on course portfolios was presented by Lee Shulman in a keynote address with a follow-up workshop session with Steve 
Dunbar (University of Nebraska-Lincoln) and William Cerbin (University of Wisconsin-La Crosse). Subsequently, the SlUE faculty roles 

and responsibilities initiative sponsored three campus symposiaon course portfolios, with SteveDunbar(i996),PatHutchings(i997), 

and Dan Bernstein (1999) as visiting scholars. During the 1997-1998 academic year, a working group of interested SlUE faculty met 
monthly to develop models of course portfolios that could be shared with the faculty. Sample course portfolios have been placed on 
reserve atthe library, and a number of those are now available online for SlUE faculty. 

In an effort to coordinate the freshman foundation art studio curriculum, SlUE’s art and design department has developed and 
instituted the use of course portfolios for each core class-basic drawing, beginning figure drawing, two-dimensional design, and 
three-dimensional design. These notebook portfolios are extensively researched reflective documents that were assembled after 
thoroughly assessing and analyzing the existing curriculum. They were designed to alleviate an unstructured situation stemming in 

large part from the ever-changing rosterofinstru ctorswhoteachthesecourses-mainly graduate studentsand adjunct lecturers. Prior 

to this, there was inconsistency and lack of cohesion regarding the material being taught. 



While recognizing that there is strength in diversity, the art and design faculty feels that it is imperative for every instructor to be 
consistent in terms of covering and emphasizing specific topics. The course portfolios are both meant to be fluid curriculum guides 
as they contain different assignments and to allow for individual approaches within each major area. Peer review sessions occur 
regularly throughout the semester. An assessment benefit of using these portfolios has been the direct comparison of student work 
produced from similar assignments from across the various course sections. Informed observations can be made, pedagogical 
strategies devised, and suggestions offered to improve course curricula, meet department goals, and enhance student learning. 



Conclusion 

William Cerbin, who was a pioneer in the development of course portfolios, made the connection between the scholarship of teaching 
and learning and assessment in an article \n Journal on Excellence in College Teaching in 1994. His concern wasthat higher education 
had made progress in looking at both student learning and quality teaching, but that the connections between those two critical 
aspects of a student’s education were generally ignored or overlooked. As he wrote: 



Assessments of learning typically document students’ knowledge but do not examine how classroom practices contribute to 
student learning. Traditional methods for evaluating teaching examine instructional practices but often ignore how those 
practices influence students’ learning, thinking, and development. (Cerbin, 1994) 

The convergence of the scholarship of teaching and learning, as sponsored by SlUE’s faculty roles and responsibilities committee, and 
the assessment of student learning, governed by our committee on assessment, has allowed us to focus on the interplay (to borrow 
a word from Cerbin) between teaching practices and student learning. At the same time, it has allowed faculty to assess student 
learning deeply and has made assessment of student learning a worthwhile and meaningful faculty activity. 
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Creating an Assessment Culture: 

A Case Study of Success and Struggles 

Colene J. Lind and Mark McDonald 



Changes in the higher education marketplace exacerbated by an economic downturn have forced many private colleges to the closing 
point. Since January 1997, 27 of 1,600 private colleges in the United States have closed, a 35 percent increase from the previous five 
years (Zhao, 2002). To survive, some institutions have changed their identity, shifting curriculum, method of delivery, and mission. 
Others have changed in name only, droppingthe provincial “college” for “university.” Our college has contemplated similar changes 
in a quest for more students, but the shift toward assessment-based improvement of student learning has been a more fundamental 
challenge. 

Our purpose is to review the special case of Bethany College in Lindsborg, Kansas. Like many small colleges, Bethany has struggled 
to design, implement, and use assessment of its academic programs. While some efforts have been successful on a small scale, 
Bethany continuesto lagin implementation of campus-wide assessment. Our efforts since the beginningofthe 2001-2002 academic 
year, however, represent a significant change. In the past, Bethany struggled to implement assessment as required by external 
agencies. Now, Bethany is working to create a self-sustaining culture of assessment. 

While there are factors that make the Bethany College experience unique, our two most fundamental challenges are common to many 
institutions. Bethany has struggled to embrace assessment as its own, and Bethany has struggled to motivate and facilitate faculty 
members in their efforts to assess and improve student learning. What we have learned from our experience may be useful to other 
regional, private, or financially troubled institutions. Our experiences may also be useful to colleges or divisions within larger 
institutions. 



Background 

Bethany is a Lutheran-affiliated four-year college, offering the bachelor of arts degree. While the college maintains a commitment to 
the liberal arts in the general education and majors offered, the majority of students graduate with professional degrees, including 
teaching, business, and social work. Located in a rural setting with few recreational opportunities, Bethany struggles to attract and 
retain students. After gradual declines in enrollment since a high of 788 full-time students in 1980-1981, enrollment has leveled off 
at about 600 full-time students peryear. 

For at least ten years, Bethany has made various efforts to implement assessment of student learning, but these efforts mostly have 
fizzled, partly because of a perception that the college has faced more pressing issues. Our most immediate challenge has been our 
tuition-driven budget, which has suffered major setbacks as enrollment has declined. Also, administration and faculty turnover has 
made any long-term project for change difficult to sustain. Since 1989# six different college presidents, five different academic deans, 
five different registrars, and five different deans of admissions and financial aid have served Bethany College. Over the past seven 
years, an average of 16 percent of the roughly forty-two full-time faculty members have been new to the institution at the beginning 
of each academic year. 

Beginning in 2001-2002, the new college president, the academic dean, and several faculty members agreed that a new approach 
was needed. Rather than treating assessment as one more thing for administrators and faculty to do, assessment would need to be 

integrated intothe day-to-day responsibilities of college employees and students. Since thistime, Bethany has tried— sometimes with 

success, sometimes not-to turn from teaching toward student learning as our college's reason for being. 



How to Change a Culture 

□ Changing Administration Priorities 

When funds do not exist, being creative is a matter of survival. For example, Bethany’s administrators agreed that adding a 
position for an assessment officer would be best. Instead, the dean gave release time from teachingto a faculty member, making 
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him responsible for coordination of academic assessment. Finding monies for training, workshops, and off-campus seminars 
also has been a creative but crucial process. 

While the do-what-we-can attitude has allowed our college to survive, we acknowledge that it cannot be a long-term solution. For 
years Bethany administrations have made generalized commitments to assessment, but without money or resources to back 
them up, these commitments were empty. We know, however, that even at the most financially strapped institutions there is 
always money. Spending on assessment simply means not spending on something else. 

But when assessment is pitted against other budget priorities, assessment usually loses. Only whenthe administration assumes 
that assessment is a regular part of doingbusiness will it receiveconsistent financial support. At Bethany, we now recognize that 
to survive, we must distinguish our college from our many competitors. As we work to define our niche, administrators are 
challenged by a lack of good information. In higher education today, it is too risky to use the finger-in-the-wind method to decide 
what type of curriculum to offer, what teaching methods are most effective, or which students to recruit. From this perspective, 
assessment is a state of being, not a corner that can be cut. 

Administrative support for and participation in a ten-member Bethany team that attended a recent AAHE-Higher Learning 
Commission assessment workshop is one example of our administration’s shifting perspective. The expense of sending so many 
to this meeting seemed huge to our dean and president, but they recognized that the cost was small compared to the price of 
another stalled assessment initiative. Designed to give participants time and resources to develop an action plan specific to our 
institution, the workshop was very beneficial. Our team developed a realistic plan to begin our cultural shift, and all ten 
participants, some of whom had had little understanding of assessment, became vocal supporters. 

□ Shifting Faculty Thinking 

For example, one long-time faculty member reported on his conversion experience, saying that he reached his epiphany when 
he realized the importance of assessment in helping his students learn-the very reason he had dedicated thirtyyears of his life 
to teaching. In general, we have found that faculty members who have experienced the power of assessment to improve student 
learning are committed to assessment. 

We are trying to recreate this faculty member’s experience. Past assessment efforts were hindered not only by a lack of time and 
money but also by the faculty assumption that assessment meant more work. Rather than presenting assessment as a 
responsibility that requires a large time commitment, faculty now are encouraged to start small. Complex pre- and post- 
assessments requiring a lengthy rubric and multiple evaluators are out. Gathering data at one or two points through course- 
embedded assessments is in. The idea is that once faculty members have small, personal successes with assessment and 
improved student learning, they will have the motivation and confidence to use assessment more extensively. 

Starting small also gives the faculty flexibility. When designingourcurrent program, the faculty assessment committee kept the 
rules simple and straightforward. We asked each faculty member to identify one or two learning objectives per course, design 
an assessment, and report the data and changes made based on the findings. Biannual reports to the committee are the only 
external verifications and paperwork. While the committee may make suggestions, we see ourselves as facilitators, not 
policymakers or the accountability police. This non-punitive and hands-off approach is intended to place power and responsi- 
bilityin the hands offaculty sothey may develop assessments that work fortheirdisciplines and unique classroom environments. 
It also gives the faculty room to experiment and perhaps fail. Our classrooms provide students with a safe place to experiment, 
risk, and, therefore, learn. We wish to provide the same environment for faculty to learn about student learning. 

In general, we have found framing assessment in terms of student learning rather than in terms of external accreditation to be 
a natural way to engage faculty. Of course, is it more efficient to simply tell faculty members that they must assess if they want 
to keep their jobs, and there may be situations that require such coercion. On a large scale, however, tough talk has failed at 
Bethany. We are now allowing people to change their own minds. 

□ Developing a Common Vocabulary 

Assuminga constructivist perspective, culture is built out of cooperative efforts to create shared meaning. At Bethany, we quickly 
realized that the strange and sometimes inconsistent language of assessment confused us. What is a mission statement? Are 
learning objectives and standards the same thing? For the first year of our renewed effort, on-campus training focused heavily 
on developing Bethany’s answers to these questions. 

Training and development also helped faculty members see the place of assessment within their accepted cultural roles. As a 
college that trains so many public school educators, for example, is it not our responsibility to model concern for student 
learning? The casual ethnographer who listens to formal and informal faculty discussions on our campus will detect the change. 
Less often do we talk about the struggles of teaching. We now hear more talk about supporting student learning. 
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□ Developing Institutional Practices that Assume Assessment 

Since creating common definitions of what assessment is and how and why we do it at Bethany, this year we have begun to 
reshape our institutional practices to integrate assessment into our decision-making system. Our AAHE-Higher Learning 
Commission team recognized that Bethany’s past efforts failed largely because assessment was tied to an individual. When the 
inspired faculty member or administrator left the college, so did all energy for implementing assessment. To incorporate 
assessment into the Bethany culture, we recognized that assessment would need to be biggerthan anyone person. It would have 
to be as big as Bethany itself. 

Early in 2002-2003, faculty approved a catalog statement affirming our commitment to using assessment to improve student 
learning. We are also changing the college’s syllabus standard to include a statement of learning objectives and assessments 
included in the course. We see these highly visible symbolic changes as important to our cultural shift. 

Our ultimategoalisto use assessmentto guide all curricular decisions, including what courses toaddorsubtract, what academic 

majors to develop or remove, and what courses to require for general education. For this to happen, assessment cannot be 
relegated to one committee. In our utopia, there will be no place for a stand-alone assessment committee or even an assessment 
coordinator. Instead, all divisions, all faculty committees, and the faculty at large will use student learning as their ultimate 
criterion. 



Future Directions and Possibilities 

For themost cynical on our campus, assessmentis still a hoopthrough which an outside agency is forcing us to jump. Thankfully, these 

individuals are now at the borders of our culture. More and more, our cultural center is defined by those of us who assume that 
assessment is part of what it means to be a teacher in the same way that preparing syllabi and grading are things that we do. More 
to the point, Bethany is developing institutional processes and cultural assumptions that will ensure that assessment continues no 
matter which individuals stay or leave. 
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Employee Development to 
Campus Development: 
Captain Assessment to the Rescue 

J. Janelle Masters and Jane M. Schrecl< 



Introduction 

Passions run high over the assessment movement, and they should. Assessment is important. Colleges and universities have a lot 
at stake in the success oftheir assessment programs. But more importantly, assessment can be powerful, harnessing energy, creative 
thinking, and information to improve student learning and otherwise accomplish good things. 

This article explores some of the challenges and opportunities that our assessment committee encountered in trying to maintain the 

viability and validity ofthecourse-embedded element of ourassessmentprogram while atthe same timeorchestratingthe expansions 
and refinements we planned for the program. Combining a clear focus on improved student learning with representation from across 
campus, we have been able to meet challenges and opportunities with creativity and a spirit of possibility best personified in our 
signature superhero, Captain Assessment. 

We hope to offer examples and experience that may be instructive and useful to institutions confronting similar situations. Just as 
importantly, we hope to encourage institutions to bring a powerful newvisiontotheir assessment programs and discoverthe potential 
for accomplishing good things throughout their institutions. 



Background 

Bismarck state College (BSC) is a two-yearcommunity college in Bismarck, North Dakota. We have a growing enrollment of nearly 3,200 
students both online and on-ground. Thirty to 40 percent of our students are in job-readiness programs, working on requirements for 
a certificate, diploma, or associate of applied science degree. The rest are in a curriculum that would prepare them to transferto a four- 
year institution. All of our programs require some general education credits. 

In the last fewyears, we have become a Learning First College. This means we have reorganized the way we do everything into a system 
that puts the emphasis on learning. All faculty and staff are distributed among six Learning First teams, each responsible for different 
aspects of running the college. 

Over the last several years, BSC has developed a program for the assessment of student academic achievement based on course- 
embedded assessment and four-point primary traits analysis scales. While we had faculty input on the program at every step of the 
process, some faculty still resist the implementation of the program. Some of the disciplines have been using a consistent system for 
several semesters, and some for less than that. Still, most faculty members are familiar with the program and use it with some 
consistency. Many of ourvocational and technical programs long ago established assessment processes based on industry exams and 
certifications or that were otherwise driven by the industry. The assessment program helped systematize these results and report the 
data from the various programs in a consistent format. 

In October 2001 we hosted a Higher Learning Commission site visit team. The team was charged with evaluating our program for the 
assessment ofstudent academic achievement, our distance education program, and the process we used to determine which courses 
were considered general education credit. Concerningour assessment program, the team was satisfied with what we were doing with 
course-embedded assessment and with plans for refinement and expansion. They want to receive a report on our progress in 
implementing the plan as we had outlined it. 

Our plans were to refine our course-embedded assessment and expand the assessment program, most notably in the area of broad- 
based assessment. We also recognized that we would have a challenge keepingthefaculty interested and serious about their course- 
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embedded assessment efforts. Along with fostering a generally positive attitude toward assessment among the faculty, we wanted 
to make our students more aware of the assessment program. Finally, we had the ongoing challenge implicit in any assessment 
program: to continue to use the results to improve student learning. 

We expected to experience a bit of a letdown afterthe preparation for and excitement of the site visit, and the assessment committee 
was looking for ways to keep the campus aware of assessment and prevent any lapsing into indifference. 



Superheroes, and Super Powers, and Super Parodies 

Dealing with the threat of indifference from the faculty and unawareness or apathy from students and staff was a job for a superhero. 
Enter Captain Assessment, complete with a cape, a cowl, and the amazing Binder of Power Assessment Handbook. Through a series 
of elaborate and imaginative comic strips in the student newspaper. Captain Assessment brought both humor and visibility to the 
assessment program at BSC. The comic strips, which combined digital photography and graphics, highlighted various aspects of 
assessment, creating a general impression of both playfulness and effectiveness. The comic strips also actively engaged the student 
newspaper staff and gave assessment broad visibility among students, faculty, and staff. 

The comic strips were able to clarify some aspects of the assessment program and blunt some thorny issues on campus, while also 
providing subtle and off-beat humor. Some of the negativity associated with assessment was dispelled through the strip. It also 
provided an opportunity to feature aspects of campus that can be overlooked or taken for granted. Some of our administrators, too 
often perceived as staid or unapproachable, appeared in the comics. The president herself figured prominently in one strip, gaining 
the ability to fly through the powers of assessment! 

One ofthe strangely encouraging signs that told us we were increasingthe visibility and understanding ofassessment on our campus 
came in an e-mail message from the Learning First team charged with building community. The message Invited staff and faculty to 
one of the team’s monthly social gatherings, which was billed as The Assessment RnR. The tongue-in-cheek Invitation included a 
parody of a four-point primary traits analysis scale, one that could be used to assess participation in campus social gatherings. Since 
it is hard to parody the unfamiliar, we on the assessment committee chose to see the invitation as an indication that assessment had 
entered into our campus culture to such an extent that it could be a springboard for humor. 



New Challenges and Inadvertent Benefits 

Is there any force more difficult to thwart than the inertia of “We’ve always done it this way”? This was the next challenge for the 
assessment committee. We needed to overthrow a tradition, and still we dared to hope that those involved would see the change as 
an improvement. This escapade required ingenuity, determination, daring-and clipboards. But people generally agreed the results 
were worth it. 



Traditionally, BSC has set aside a day for employee development, a day when classes do not meet, duringwhich the staff and faculty 
mingle socially as well as attend workshops together for increased camaraderie and professional development. In spring 2002 the 
assessment committee decided that it would be a good idea to include assessment activities in employee development day. At the 
same time the state mandated giving a standardized student satisfaction survey to the student body, and our stated goal was 100 
percentcompletion.Talkshiftedtothebest wayto accomplish this task. Additionally, a common and annual complaint amongfaculty 
and student services was the dearth of faculty adviser-advisee interaction; there did not seem to be any time set aside for advising 
activities, and many students planned their schedules alone, often making errors that delayed their graduations. 

After much discussion, it was decided that employee development day on Monday, October 21, would be transformed to campus 
development day on Wednesday, October 23, a strange new hybrid event for students, faculty, and staff. The morning would be used 
for students to meet in a group with thelradvlsers. It was conceived as a timeforgeneral advising information, assessment awareness, 
and the administration of the student satisfaction survey. The afternoon would be the time when all employees would meet together 
offcampus for social and professional activities. Furthermore, byworkingwith the learning innovations team (the Learning First team 
charged with planning employee development day), we arranged to have assessment be the topic ofthe keynote address in the 
afternoon, as well as one ofthe breakout session choices. 

Transforming employee development day to campus development day took massive planning. At first it was thought that the plan was 
loglstically impossible, that there simply was not enough space on campus to gather more than 3,000 students together. We weren’t 
even sure we had enough rooms for all of our advisers. 



Er|c 



Lists of advisers, advisees, and classrooms were gathered and scrutinized. A walking tour ofthe campus, complete with clipboards, 
verified that we had more than enough classrooms for advisers, but that the seating capacity in most rooms wouldn’t accommodate 
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all the advisees at once. Two sessions were planned, with halfthe advisees at 9:00 a.m. and the other halfat 10:15. The logistics of the 
adviser-advisee sessions were worked out with only a couple members of the faculty being displaced from their regular rooms. The 
Learning Innovations team drew up lists of talking points to assist advisers during the sessions. With all this preparation, we felt that 
the advising sessions could work as long as we could get the students to come. 

Students were used to having this day off. We expected confusion, and we expected resistance. We needed a plan startling in its 
irregularity, personal in its appeal, and seductive in its bribery. We decided to produce a letterto each advisee and to have the students 
individually pick up their letters in the library. 

One intrepid and trustworthy work-study student was given the daunting task of typing advisers* letters to all students. Fortunately, 
he was armed with the superpowerofmailmerge.The letters gave the time and placefortheadvisingsessionand promised incentives 
for attending: students would each receive three discount coupons to the bookstore, and their names would be entered into drawings 
for a free faculty-staff parking pass and a $200 gift certificate to use in the bookstore for textbooks. 

The letters were placed at the front desk of the library under the names of advisers and were handed out by the library staff. Each 
students needed to know his or her adviser's name or could look it up on a list posted in the hallway. Two weeks ahead of campus 
development day, students were reminded by instructors to pick up their letters and come to the advising sessions. The letters that 
weren't picked up were mailed to students. To further promote the event, posters were developed, reminding students to come and 
promoting assessment in general. 

Some administrators were sourly predicting a 30 percent turnout; some cockeyed optimists were hoping for 75 percent. Considering 
that there were no direct consequences for not attending and that about 20 percent of our students are part-time, many taking only 
evening classes, most people seemed satisfied that over 50 percent of students attended their advising sessions. 

Campus development day provided several expected benefits beyond those directly related to assessment. Not only did we feel like 
we had made a very good effort in complying with the survey mandate, but we also felt that the morning had helped to strengthen the 
adviser/advisee relationship-some students learned their advisers' names for the first time. We had advanced the cause of 
assessment in the morning with the students and in the afternoon with faculty and staff. Furthermore, the promotional posters were 
displayed all over campus for days ahead of the event, reinforcing assessment issues for faculty, staff, and students. 

There were also several unexpected benefits. Going to the library and being handed their letters by a real, live librarian can only serve 
to make students more likely to come to the library again. Also, the bookstore has enjoyed increased traffic since students received 
their discount coupons. 

Beyond that, we had turned the battleship of tradition, transforming employee development day to campus development day with 
barely a ripple. In the evaluation surveys filled out by employees after campus development day, most agreed that half a day of 
development activities seemed like enough. Ofcourse, improvements were suggested, but faculty members were almost unanimous 
in their approval of the concept of advising sessions, and many staff members seemed to appreciate a morning to catch up on work 
out of the routine of a regular school day. 



Unexpected Treasures Discovered While Exploring Other Stuff 

One of the important expansions of our assessment program was to develop valid, broad-based measures for assessing our students' 
learning in the general education curriculum. One of the things that seemed important in assessing the effectiveness of our general 
education curriculum was to determine how well students are able to apply whatthey learn. While course-embedded assessment has 
many advantages, experience and anecdotal evidence about students unable to write in biology, speak in history, or interpret data 
in sociology suggested thatassessingskllls In the courses in which they are taught might not be the most valid way of measuring what 
our students are learning. How well students performed in a composition class, for example, was not necessarily a valid indication 
of how well they would be able to write outside of that class. We were seeking a valid, feasible way of assessing knowledge and skills 
in courses outside their discipline. 



ERIC 



Amongthe ideas thatwe explored were capstone courses and cross-disciplinary learning communities. Capstone courses seemed like 
a natural opportunity to gather assessment data on students' ability to pull together and apply what they have learned; however, no 
capstone courses existed In our general education curriculum or in the expectations of our students. As noted earlier, students might 
be working on a certificate, a diploma, or an associate of applied science degree through our job-readiness curricula or an associate 
of arts or associate of science degree in our transfer curricula. General education requirements vary from four semester hours for a 
certificate to nine for a diploma, fifteen for an AAS, and thirty-six for an AA or AS. Given this variety of possible profiles in general 
education enrollment, implementing appropriate capstone courses did not seem feasible. 
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Cross-disciplinary learning communities seemed more workable, and also advantageous for reasons beyond the assessment 
program. These learning communities have developed in a couple of different ways on our campus. For some, we yoke two or three 
courses together, and students enroll in some or all of the particular sections of those courses, knowing that there will be a built-in 
cohort of students and shared material and discussion among the courses. The courses are still separate and still maintain the 
common course numbering and credits of our system, but they are linked by theme and enrollment, creating a learning community. 
The other form of cross-disciplinary course that has developed on our campus is a general education course for particular kinds of 
majors, specifically a speech course for science majors. 

While the assessment committee and some administrators continue to encourage the development of these cross-disciplinary 
opportunities for students, they haven't yet become a significant enough part of the campus culture, with students or with faculty, to 
provide valid data for broad-based assessment. 

Still, these learning communities are worthy of mention because they are an example of the unexpected treasures that can be 
unearthed when on a quest for something else. Even if we are not yet able to make use of these courses for broad-based assessment, 
they are a positive educational opportunity for students and faculty alike, and their development is being encouraged and promoted. 



Making Assessment a Way of Life 

The quest for a valid method of broad-based assessment of our general education curriculum continued. Simultaneously, experience 
taught us that some refining and revising of our general education objectives was needed to make them more easily assessable. We 
had been working with ten general education objectives for a number ofyears. These were developed in conjunction with a rewrite of 
our college's philosophy and mission statements. Most of the ten objectives had specific subpoints; some objectives were still ill- 
defined. The assessment committee took on the task of examining these objectives to ensure that theywere all assessable and were 
truly being assessed. 

This revision processturned out to begrueling, exhausting, surprising, and finally rewarding. At one stage in the process, the ten main 
objectives became forty-two assessable subpoints. Further examination allowed for some sensible combining and eliminating. The 
ten main objectives remain, but for the purposes of assessment we now have twenty-four assessable points or intended outcomes 
within them. 

Oneweekend, the assessment coordinator for our college-a poet by training and an administrator by experience-was struggling to 
pull together a workable proposal for broad-based assessment of general education. The assessment committee was scheduled to 
meet on Monday, and she wanted to have a proposal to consider at the meeting. She was working at her dining room table-soothed 
by Gregorio Allegri's “Miserere mei, Deus” on the stereo; intrigued with the majesty of the “V” of Canadian geese flying overhead 
outside her window; comparing and synthesizing various tomes on assessment; balancing and integrating our college's particular 
needs, hopes, and constraints-when it struck her: “This is general education.” Art, science, geometry, reading, writing, critical 
thinking, problem solving, research skills, evaluation— even acknowledgement of diversity— at that moment, she was making use 
of it all. 

Education has to do more than teach students to write in a writing class and manipulate numbers in a math class. Education has to 
enable students to learn when and how to use their knowledge and skills effectively in any situations in which they find themselves. 
Because ofthat, valid assessment programs have to do more than assess writingin a writing class and math in a math class. They have 
to be cross-disciplinary; they have to be systematic; and they have to be systemic. 

The result of her insight that morning was the spark of an idea that was brought to the assessment committee that Monday and 
eventually turned into our four-year plan for broad-based assessment of general education. This is a plan that combines standardized 
testing, program assessment, cross-discipline assessment, and many of the processes we have learned through course-embedded 
assessment. It sets the assessment activities up on a rotation that ensures that each of the twenty-four assessable points will be 
assessed on a periodic basis. It spreads the assessment workload among the disciplines and faculty groups. It uses multiple 
measures. And. finally, it has a significant cross-disciplinary element. 



Summary 



For those who think the drama of this case study is overstated, we'd like to declare that assessment should be an adventure. It's easy 
to become bogged down in the paperwork and negativity that can surround assessment programs on college campuses. We hope our 
story can inspirethoseinvolved with assessment to bringanewspirit ofcreativity and possibilityto their efforts. What superhero would 
help reanimate an assessment program that may be stalled from neglect or familiarity? What super powers are needed to subdue the 
forces of evil bent on foiling assessment on your campus? 
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Our experience tells us that assessment programs that remain focused on improving student learning while also staying open to 
challenges throughout the institution can accomplish a great deal and benefit the institution in unexpected ways, using their super 
powers for good-the good of im proving student lea rning. 

To paraphrase another superhero, Learning, Teaching, and the Assessment Way! 



J. Janelle Masters is Assistant Dean of Instruction and Chair of the Arts and Communications Department at Bismarck State College 
in Bismarck, North Dakota. 
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Creating a Culture of Assessment: 
Empowering Faculty to 
Access Assessment Data 

Susan Hatfield and Theresa Waterbury 



While many institutions have developed processes and procedures to implement assessment activities on their campuses, few have 
evolved to the point where they have created— almost ten years into the initiative-a culture of assessment. 

What^s taking so long? 

Perhaps one of the issues significantly inhibiting the creation of a culture of assessment is the “path of least resistance” approach 
adopted by many administrations in order to implement assessment on their campuses. Recognizing that they had fallen behind in 
the implementation strategies set forth in the assessment plan submitted to the Higher Learning Commission backin the mid 1990s, 
campuses scrambled to catch up as their reaccreditation visits approached. Low-impact assessment strategies were implemented, 
essentially urging faculty and staff to “)ust Do lt”-do anything-to appease the Higher Learning Commission and the site visit team. 
The implicit agreement was that as soon as the team left, faculty could return to business as usual. 

This “Let’s Make a Deal” approach generated data for the self study and resource room, though it usually resulted in follow-up reports 
and focus visits on assessment. 

Winona State University (WSU) has adopted a culture-building approach to our assessment initiative. WSU was aware that building 
a culture would be a time-consuming evolutionary process, but the eventual result would be one that would create a long-lasting 
commitment to assessment. 

Our early efforts to create a culture of assessment were through education and incentives. Educating faculty and staff created a 
common language from which to discuss and debate. Financial incentives were given to the departments and programs that 
volunteered to be early adopters. While both of these strategies began to establish the foundation of an assessment culture, it wasn’t 
until we empowered faculty to access data in real time through the development of a Web interface to our database that they really 
began to understand and embrace assessment. 

As in most colleges and universities across the country, Winona State’s assessment activities were severely limited by the inability 
to efficiently and effectively store and access data. Data were stored in collection silos, with a separate silo for each type of data 
(student survey data, financial aid data, admission data, student records). These silos were located in different offices throughout the 
campus (admissions, registrar, housing, and academic affairs) and had no interconnectivity as each database came online at a 
different time and without universal data collection standards. Compounding the problem was that in many cases data dictionaries 
were nonexistent and data definitions resided only in the minds of those few who had actually entered the data. The result was that 
much of the existing data was unlnterpretable (except by a few individuals who knew what the data elements represented), and could 
not be linked together. The ability to explore relationships was, for all practical purposes, impossible. 



Er|c 



An improved technological Infrastructure was required. The rationale was multifold. First, creating a structure— as opposed to a 
procedure-would be more likely to have a long-lasting impact on campus and really move us toward the creation of a culture of 
assessment. We knew that we could quickly and easily design assessment procedures that might be impressive to a site visit team 
whose members were not very astute in assessment, but procedures alone would not have the cultural impact as a structure for data 
collection and analysis. Secondly, a solid technological infrastructure would move some assessment activities away from the 
department level to the university level, allowing departments to focus on student learning issues. Measuring issues such as study 
habits and student satisfaction and many indirect measures of student learning can be easily done on the university level, freeingthe 
departments to focus their energies on direct measures of student learning. Third, a smartly designed technological infrastructure 
would position us to respond to relevant questions for either a traditional reaccreditation self-study or, should we choose, an AQIP 
portfolio. Finally, the creation of a robust technological infrastructure would allow faculty and staff to see assessment in action, and 
thus would create a sense of understanding, interest, ownership, and investment in assessment. 
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The technological infrastructure that we created with the assistance of a U.S. Department of Education Title III Strengthening 
Institutions grant consisted of an integrated database and four software applications—a student information tool, a faculty 
information tool, an ad hoc query tool, and a report tool. The first three applications are not assessment tools but information tools, 
as they query existing university records. We began with their development in order to have faculty and staff become familiar with the 
software by providing them direct access to the kind of database information that they would routinely request from the institutional 
research office. These applications were first introduced to office managers and administrative assistants, who offered invaluable 
assistance in their development. 

The report tool application, completed and introduced to the campus in August 2002, allows secure access to student self-report 
survey data (behavioral and observational) from our Web-based student survey modules. These survey modules, completed by 
students every spring as part of our assessment day activities, consist of a series of question modules related to individual behaviors 
(social behaviors, study habits), background, perceptions (quality of instruction, campus climate), and satisfaction. When students 
login to the assessment Web page, they access specific modules depending upon the number of credits they have earned, their major, 
and their home campus. In addition to the university-wide survey modules, several departments and programs have included 
questions specifically designed and delivered only to their majors. As a result, a second-semester sophomore nursing major would 
receive a different set of survey modules than a second-semester junior biology major. 

The reporting tool also allows access to department learning outcome data, which are also included in the database. For Instance, if 
a department is using a standardized test as a learning outcome measure, those scores can be entered into the system and are 
available to be accessed along with the student self-report data. These self-report data provide faculty with a context in which to 
understand outcome data, adding life and meaning to a set of results. For example, a list of test scores on the CAAP exam becomes 
more interesting if analyzed in relation to how the scores of first-generation students compared to the scores of students for whom 
higher education has been a family tradition. 

Unlike the other applications we developed, which report line-item data, the reporting tool reports aggregate data for the user- 
specified population. Controls are in place to protect student anonymity. Specifically, if the population or comparison group selected 
contains fewer than five students, data are not returned. This prevents the identification of individual students by gender, race, and age. 

The reporting tool allows the user to identify a specific population (by year, college, department, or campus) to be studied. Following 
the identification of the population, the user identifies the process (self-report) module or outcome module as the object of analysis. 
As data only have meaning in relation to other information, system users have the opportunity to identify two levels of comparative 
variables, including background and demographic information. As such, the user may wish to compare study habits for upper-division 
students to those of lower-division students for each of the past three years. Or they may investigate whether female students who 
study more than fifteen hours per week perform better on an outcome measure than males who study less. Users can also identify up 
to four aggregate numbers to be reported for each response quadrant, choosing from the number in each response quadrant, column 
percent, row percent, total percent, or mean of ACT of the population that reported in that category. This allows for a single report to 
contain a significant amount of data for the users in their investigation of their research question. While not proving causality, the 
report does allowfor the investigation of correlations between processes and perceptions and learning outcomes, providing for richer 
interpretation possibilities than would be possible from studying process or results data in isolation. 

Faculty and staff training on the report tool began in late August 2002. Since then, more than 150 faculty and staff members have 
participated in the training and are regular users of the system. 

We have some observations about how the reporting tool has assisted in developing a culture of assessment. 

1. Once faculty members learn how to access the report tool, they are asking increasingly complex questions related to our 
students* learning. 

2. Faculty members are starting to recognize the importance of sample size and are strongly encouraging— in many cases, 
requlrlng-students to participate in assessment activities. 

3. More departments are providing outcome data for Inclusion in the database because they want to be able to analyze the data 
using the online report tool. 

Our report tool has played a critical role in further developing Winona State*s culture of assessment. Enabling faculty to access 
assessment data in an easy, convenient, and intuitive way has promoted their interest, understanding, investment, and ownership 
of assessment. 



Susan Hatpeld is Assessment Coordinator and Title Hi Project Director at Winona State University in Winona, Minnesota. 
Theresa Waterbury is Data Analyst Specialist and Title III Activity Director at Winona State University in Winona, Minnesota. 
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Engaged Learner and Assessment: 
A Faculty-Student 
Shared Responsibility 

Jian R. Sun and Greg Miller 



Introduction 

Student outcome assessment is no longer a strange concept to institutions of higher education. The initial discussion on assessment 
in NCA*s 1989 documentation has prompted significant institutional changes in awareness, attitudes, practices, policymaking, 
implementation, and institutional structures. What we perceive to be significant in this movement to make assessment an integral 
part of an institution’s mission is how assessment is being implemented at different levels. We are moving from viewing faculty and 
students as two separate entities with different responsibilities in the assessment process, to engaging faculty and students as a 
unified entity responsible for: designing the process, generating assessment content, assessingthe collected data, and interpreting 
and analyzingthe information. 

This paper provides information that can help address four basic questions: 

1. What does “engaging learners” mean relative to assessment? 

2. What is the role of faculty in this engagement of learners? 

3. How can this engagement be reflected in the design, implementation, and interpretation of assessment activities and 
subsequent results? 

4. What are some difficulties and challenges? 

In this report we provide a description of the assessment component that is being modified to emphasize the design of assessment 
activities. We want togenerate meaningful assessment information at three levels: course, program, and institution.We need to make 
the linkage between the pedagogy and practice for the teaching and learning process. We believe that there should be continuous 
discussions of rationale behind such modifications, advancing current levels of understanding assessment. 



Discussion 

The main thesis forthis report is that assessment should be viewed asa shared responsibility between faculty and students. Ourthesis 
is supported in each of the following three broad domains: 

o Teaching philosophy/pedagogy concerning the role of the learner and the role of the teacher in the teaching and learning 
process 

o Best practices addressing the relationship between assessment and teaching and between assessment and learning 
o External expectations from professional, state, regional, and national accrediting organizations 

Significant amounts of literatureon teachingand learning have touched upon the roles ofthe learner and the teacher, describingthem 
as complementing each other (Dawson, 1997), in termsofthe learning styles that learners adopt oradapt (Erwin, 1991; Gardner, 1993), 
the teaching practice by the teachers (Miller, Imrie, & Cox, 1998), the discipline characteristics that may impact the teaching style 
(Huber 81 Morreale, 2002), the delivery methods used (Brooks 81 Brooks, 2001), and the evaluation theories of learning (National 
Academy of Sciences, 2001). 
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Best practices in assessment, guided by those various pedagogical implications, have been providing more and more information. 
This information focuses on the two types of relationships in assessment and teaching and learning, as is best described in the 
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“assessment triangle” concept discussed in a recent report by the National Academy of Sciences (2001). This report details three key 
elements underlying assessment; “a model of student cognition and learning in the domain, a set of beliefs about the kinds of 
observations that will provide evidence of students’ competencies, and an interpretation process for making sense of the evidence” (p. 44). 

It is now hardly deniable that external expectations of various constituencies have played a significant role in providing both pressure 
and guidelines in requiringinstitutionsofhigher education to more actively engage the learner in the process ofassessingthe learning 
outcomes. Such pressures and guidelines have resulted in knowledge and understanding of how students learn, how teaching can 
become effective, and how learning can be assessed. This knowledge and understanding leads to new domains, as illustrated in 
published documentation from regional accrediting bodies, such as the Higher Learning Commission’s “Levels of Implementation” 
(2002a) and recently proposed “New Criteria for Accreditation” (2002b). 

Those newdomains have created fields where “men and women...actually practice a domain’s procedures” (Gardner et al., 2001, 24). 
Professionals have been trained to work specifically on those tasks within a field (assessment of learning) for a common purpose. Here 
is the important point for the field of assessment of learning! Students should serve one of three major roles (in addition to elite 
gatekeepers and expert practitioners). Unfortunately, this point remains unrecognized by many assessment practitioners. Students 
are involved in only a limited way in the assessment process. This limited involvement can easily jeopardize the desirable outcomes 
of assessingstudentlearningby creating “misalignment between a domain and a field”(Gardneretal., 2001, 30). The need to maintain 
the alignment has become essential, as can be seen in the two Higher Learning Commission publications mentioned. The emphasis 
in doing assessment of student learningwhile actively engagingthe learners, throughout their learning process, further manifests the 
newly raised issue of the importance of involving the learners at a higher level of learner engagement in assessment. 

To what extent can this learner engagement in assessment be defined and/or understood? Past and current practices seem to suggest 
several levels ofengagement.Creatingawareness of assessment in students through publicly articulated statements of assessment 
activities to be completed by students serves as one level. Students should receive information about assessment at the outset of the 
learning process. The role of the faculty is to educate students about assessment. At this level, the education of students about 
assessment is important, and it is referred to as being important for the institutional culture (Higher Learning Commission, 2002a). 

At another level, learner representation in many of those assessment organizations is encouraged, and the learners are constantly 
motivated to perform required assessment activities (whose format and content are pre-decided and pre-designed by faculty, not by 
students) bytakingvarious tests, respondingto surveys, and completingactivities and projects intended togenerate assessment data 
to be analyzed and assessed by faculty. The faculty’s role can be found in the faculty’s efforts to embed assessment in their course 
assignments. At this level, the learners are the sole providers of raw assessment data, to be interpreted and analyzed by others. This 
level describes part of the s/7G/'et//'espor7sM//y (Higher Learning Commission, 2002a) with students in the assessment process. 

What is advocated in this presentation is yet another level, where the learners are given the charge of selecting and presenting the 
content of their learning outcome evidence, and assessing such evidence, to demonstrate their learning outcomes. With the help of 
the faculty, students are offered the opportunity to select samples of their documented learning and provide their own assessment 
information. Faculty members serve as facilitators and engage the students, during the assessment process in each course, in both 
selecting and assessing their learning samples. In this process students are maximally engaged in the assessment process through 
a redefined division of labor for assessment, referred to as maturing stages of continuous improvement in sharing responsibilities 
with students (Higher Learning Commission, 2002a). 

Those three levels of engagement seem to characterize the multiple roles of the students as “information receivers,” “passive 
providers of assessment data,” and “active assessors of their own learning.” Those three levels characterize the faculty as “mere 
providers of assessment information,” “reinforcers of assessment activities,” and “facilitators in the assessment process and product.” 

Assessment as a Shared Responsibility 

A student’s role in the assessment process should go beyond the standard student representation on campus assessment 
committees. It should go beyond being informed of assessment purposes. As stated in the recently proposed “New Criteria for 
Accreditation,” one of the three primary elements of effective learning environments should be engaged learners. What has been 
taking place in our graduate program is a continuous and progressively challenging faculty discussion on howto maximally engage 
learners in assessing their learning. From its inception our program was dedicated to seeking ways by which students could 
demonstrate genuine understanding resulting in functional knowledge. We expect students to maintain journals that document 
reflective thinking on personal, pedagogical, and philosophical levels. We expect classes to culminate in personal performances by 
each student to demonstrate genuine understanding. 

To place our graduate program in alignment with the concepts of truly incorporating assessment in the teachingand learning process, 
it has been decided that the way assessment is performed should be modified. Learner engagement, therefore, is becoming a starting 
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foundation for us to reexamine our assessment practice as designed ^implemented, interpreted, and utilized. The product from such 
conversations among faculty in our graduate program is to be shared below. 

Student electronic portfolios, as a learner-engaged approach to assessment, share the following characteristics: 

0 Its purpose; To provide evidence of student learning and to provide opportunities for recognizing needed program 
improvements. 

0 Its format: Electronic portfolio. 

0 Its three content components: 

1 . Component One (Introduction): Serves as an introduction to each student’s portfolio, where the student can include 
biographical information (name, work, career, and other relevant information of his or her choice). 

2 . Component Two (Self-reflection/evaluation): Serves as self-reflection, where students provide a realistic assessment 
of their own learning based upon selected coursework relative to the program goals and objectives. This is the most 
important manifestation of students’ learning that will be evaluated by faculty from the program as well as by outside 
evaluators. 

3 . Component Three (Documented evidence of learning): A collection of student coursework (activities, assignment, 
projects, writing, etc.) that provides evidence of student learning. When students complete all required work in each 
course, one or more work examples will be submitted after being evaluated by the course instructor and archived into 
a database. Students will have access to all the archived coursework they have submitted, and will compose their self- 
assessment piece ofwriting for the completion of the portfolio. 

This documented evidence of learning, from the course archives, will be inserted into the final portfolio, along with the 
introduction and self-reflection/evaluation. 

Four types of course documentations are to be included: 

o Effect on me (to be written at the entry, middle, and exit points) 
o Effect on my teaching (addressing issues in pedagogy) 

o Effect on my students’ learning (manifesting how knowledge and skills learned are applied) 
o My content knowledge (demonstrating content mastery) 

0 Two portfolio evaluations; 

o Internal evaluation: Facultyfromthe program will evaluate randomly selected portfolios toassesshowwellthe program 
goals are met through students’ work. 

o External evaluation: Outside evaluators will be invited to evaluate randomly selected portfolios to assess how well the 
program goals are met through students’ work. 

0 Evaluation criteria/rubric: A set of criteria, to be developed and shared with all faculty and students, will serve as the basis 
for portfolio evaluation. The evaluation will focus on the second component (self-reflection/evaluation) of the portfolio. The 
third component of the portfolio is consulted by the evaluators only when they wish to confirm or verify their analysis and 
evaluation of Component Two. This method of primarily evaluating Component Two in the portfolio serves several purposes: 

o Each student bears the main responsibility of producing and presenting evidence of his or her learning in constructing 
the self-reflection/evaluation piece. 

o Each student needsto understand the relationship between Component TwoandComponentThree.Adeep understand- 
ing and analysis of his or her coursework should lead to a well-thought-out self-reflection/evaluation. 

o There is a difference between indicating the completion of coursework and producing and presenting evidence of 
learning. 

o Assessment is not something new or added on, but is an integral part of the teaching and learning process. 

0 Dissemination: Three basic uses: 

o Serves as a capstone of student learning to collect and summarize the learning experience, 
o Provides a primary source of documentation of student learning, 
o Offers a showcase of students’ graduate educational experience and ability. 

O 
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From the descriptions and discussions presented so far, the first three questions raised earlier have been answered-namely, true 
learner engagement should refer to such engagement at a higher level, as is clearly described in the “Levels of Implementation,” with 
the faculty serving as active facilitators, helping and guiding the learners in this engagement in various phrases (design, implemen- 
tation, and Interpretations) of the assessment process. While trying to maximally utilize the assessment information for multiple 
purposes (generating assessment at course level, program level, and institutional level), faculty would need to be aware of, and 
prepared for, possible trade-offs in designing such assessment. Theories and practice have demonstrated that “the more purposes 
a single assessment aims to serve, the more each purpose will be compromised” (National Academy of Sciences, 2001, 225), and 
faculty are being challenged to explore more ways of balancing the relationship between the desire to design assessment activities 
for multiple purposes and “designing multiple assessment activities to provide the various types of information required at different 
levels of the educational system" (National Academy of Sciences, 2001, 225). 

At the local level within our program, specific challenges confronting our faculty are providing appropriate faculty development, 
creating better systems to connect the learners in the program for their engagement, strengthening the current mentoring system, 
collecting and analyzing the data generated from the portfolios, utilizing such Information to Improve teaching and learning, and 
reassessing the proposed portfolio assessment design and practice. 
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From Crisis to Culture: 
Inspiring, Cultivating, and Guiding 
a Culture ofAssessment 

Rebecca Wojcik, Eric Martin, and Joyce Kennedy 



Introduction 

Like many schools, Governors State University (GSU) attempted to “do” assessment for many years without either taking the time or 
committing the resources to establish a culture of assessment. As a result, assessment was a top-down initiative that started, 
stopped, and restarted with each new administration. Many administrators, faculty, and staff now realize the importance of 
establishing a culture of assessment that weaves assessment and continuous quality improvement into the fabric of the institution 
so that each can be sustained and strengthened over time. Learning to overcome motivational and organizational barriers that often 
impede such work is important to developing a culture of assessment. As a result of n umerous process en hancements and the central 
efforts of the university assessment committee (UAC), assessment is now a daily activity and ongoing priority. 



Developmental Stages 

To appreciate the emerging culture of assessment at GSU, it is helpful to review several key points in the recent history of the 
institution. In the early 1990s, assessment of student learning outcomes was conducted by faculty in a highly decentralized process 
with great variability of expertise and inconsistency. While several faculty members were involved in widely dispersed development 
of their individual expertise in assessment, including participation in developmental efforts at the national and regional level in the 
higher education community, most were underinformed, passive participants in assessment of student learning. Assessment 
activities were not well documented, and, at an institutional level, the scope of activities was difficult to discern. 

This situation was largely a result of widespread administrative turnover throughout the decade, and it occasioned a focused visit by 
the North Central Association of Colleges and Schools (NCA) in 1997. During this visit, NCA expressed serious concern about the 
unevenness of outcomes assessment across programs as well as the lack of assessment documentation, particularly at the 
institutional level. Thus, it was widely anticipated that assessment would be a central consideration during the university's April 2000 
reaccreditation visit. Yet the university's assessment efforts remained troubled between 1997 and 2000. In 1999, only a year prior to 
the visit, both the president and the provost resigned. The president had been with the institution for seven years, while the provost 
had been the sixth person to have occupied the office within nine years. What is more, it was revealed in August 1999 that the 
university's assessment com mittee-a group formed in response to the earlier focused visit-had not been meeting. Thus, the degree 
to which the university's assessment plan was being followed was unclear. Inside observers characterized the development of 
assessment at GSU throughout this period as episodic, driven in large part by external timelines and requirements. While overt faculty 
resistance to assessing student learning was negligible, collaborative faculty-to-faculty and faculty-to-administrator efforts were 
underdeveloped and at best isolated. 

As the April 2000 NCAsite visit approached, the interim provost reconvened the university assessment committee, but the new group 
had little understandlngof its charge. The university's newdirector ofwritingacross the curriculum, who reported to the provost, was 
included in this committee and facilitated the group's renewal and focus on development and documentation of assessment efforts. 
Atthesametime,GSU'sboardoftrustees— a group thatwasalsoundergoingtransformationduringthis period-mobilized in fall 1999 
and quickly initiated a search for a new president. The search was concluded with the appointment of Stuart 1 . Fagan, the university's 
fourth president, on April 1, 2000, just twelve days before the NCA team's arrival. 

As expected, the accreditation team identified outcomes assessment as a primary area of concern, although the team discovered that 
far more outcomes assessment was taking place in individual programs, divisions, and colleges than was being documented at the 
university level (a reflection of the widespread administrative turnover that characterized the period). After a careful review of the 
assessment processes within each of the university's four colleges, the team located GSU at Level 2 of the Levels of Implementation, 
and required a progress report on assessment in three years. 
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Process Enhancements 

Through the subsequent stabilization of the administrative ranks as well as renewed awareness of the exemplary assessment work 
alreadytakingplaceon campus, many process enhancements related toassessment have occurred at GSU since springzooo. Perhaps 
the most important enhancement was the revision of the university’s mission statement and strategic plan in 2000-2001. Duringthe 
process of revising the mission statement and plan, the strategic planning committee, president, provost, and board of trustees 
endorsed movement of the institution to “demonstrable excellence” and continuous improvement. The president, particularly, has 
consistently echoed the message of quality academic programs. 

Another process enhancement that has led to the recent success of assessment at GSU is the reactivation of the university assessment 
committee under the aegis of the provost. This appointed committee is a workinggroup, and not exclusively a thinktank. The efforts 
of the UAC have expanded in the past several years and are facilitated by the relatively small size of the institution’s faculty and the 
spirit of collaboration that has developed between the provost’s office, UAC, and the faculty at large. Unlike its predecessors, this 
revitalized committee has developed a mission statement and objectives for its work in addition to collaboratively creating a new 
model for assessment at GSU. Through retreats, brown-bag sessions with faculty, and monthly reports to the faculty senate, the UAC 
has fostered assessment dialogue on campus. The committee has successfully empowered programs by validating assessment 
activities, allowing greater flexibility in its expectations, and accepting reports on assessment that are couched in larger self-study 
documents prepared for specialized accreditations. The power of small groups working on idiosyncratic assessment issues has been 
harnessed effectively, resulting in a foregrounding of assessment and networking of ideas. 



Timing and Resource Issues 

In addition to the aforementioned improvements, several timing and resource issues appearto have contributed to the emergence 
of a culture of assessment at GSU. Throughoutthe later half ofthe 1990s, the university had an annual cycle of assessment in which 
reports were submitted to the office ofthe provost, but feedback on these documents was limited, if it occurred at all. The office 
was simply too short of staff and consistent leadership to respond. Moreover (and in large part due to staffing issues in the 
provost’s office), the format ofthe reports was prescribed as a “one size will fit all,” which further discouraged program assessment 
coordinators— faculty members charged with assessment in each unit-from completing materials and submitting them on time. 
Curiously, during this same period, adherence to the timelines and procedures established by external accrediting bodies (for 
programs that had them) was a high priority. However, because those timelines and procedures rarely paralleled those ofthe 
university’s own assessment initiative, and because assessment was not central to most external self-study processes at the time, 
internal work on assessment was given minimal attention. Such timing issues forestalled the development of a culture of 
assessment throughout the decade. However, the university assessment committee’s modification of timelines and formats to 
allow more flexibility and support for assessment as a cyclical, formative process instead of a time-driven process bearing all of 
the trappings of a summative evaluation has increased both the quantity and quality of responses from Individual programs, 
divisions, and colleges. Timely administrative feedback as well as a concerted effort to synchronize external accreditation and 
program review processes have also contributed significantly to the emergence of a culture of assessment at GSU. (These will be 
discussed further in the next section.) 

In addition to resolving significant timing issues, resource concerns are also now being addressed. The resources designated for 
assessment activities at GSU are housed in the individual programs, divisions, and colleges as well as in the central administration. 
The overall availability of resources is modest and generated exclusively through institutional funds that are sensitive to variability 
in state appropriations. Prior to 1999 no release time was provided to any faculty member to address assessment. The Center for 
Quality was established in 2001 to oversee university assessment in addition to coordinating GSU’s program review process for the 
Illinois Board of Higher Education, assisting programs with specialized accreditation and licensure requirements, and guiding other 
university quality initiatives such as faculty development and writing across the curriculum. While the budget for this unit has 
remained flat, prior to the formal establishment ofthe center in 2001, the money was used almost exclusively to pay for the annual 
Productivity, Quality, Priorities (PQP) program-a negotiated and highly controversial faculty development activity that focused on 
generic topics. Although PQP remains a contractual obligation, the renewed stewardship ofthe quality budget has placed its focus 
on outcomes assessment and related quality initiatives. In addition to the purchase of books, subscriptions, and teleconference 
registrations, investment in faculty such as support for attendance and presentations at national, regional, and state assessment 
conferences has become standard procedure. Likewise, the Center for Quality has purchased assessment tracking software and now 
sponsors training sessions on this software in addition to the aforementioned events, ranging from full-day retreats to informal brown- 
bag meetings. 

While the extrinsic rewards forfacultyto assess student learningoutcomes remain limited, the establishment ofthe Center forQuality 
and subsequent expenditure of resources to support assessment in sustainable ways has created positive change by illustrating the 
administration’s commitment to assessment. This, in turn, has fed the faculty’s intrinsic motivation to participate. 
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Related Institutional Quality Processes 

When the new university president and provost initiated a comprehensive review and subsequent revision of the university’s mission 
statement and strategic plan only weeks after receiving reaccreditation from NCA in 2000, it became clear that a paradigm shift was 
underway. The goal of demonstrable excellence was quickly identified, and with it came a related commitment to continuous 
improvement. The Center for Quality promotes both concepts and works with faculty members and administrators to foster dialog 
throughout all assessment and self-study processes. The university assessment committee carries this same mantle. Throughout the 
prior decade, faculty at GSU were minimally responsive to outcomes assessment and viewed feedback as potentially judgmental and 
punitive. Today, thanks to the changes discussed throughout this essay, increasing numbers of administrators, faculty, staff, and 
students are aware that good assessment is formative and ongoing. When the university assessment committee and office of the 
provost (via the Center for Quality) began to offer feedback and encouragement on assessment activities as opposed to passing 
judgment, the momentum of faculty efforts increased. The work of assessment is now transparent, collegial, and ultimately useful. 

Likewise, the university has benefited greatly from the aforementioned consolidation of timelines and processes imposed by external 
agencies. The assessment of student learningisnowthe linchpin for program review forthe Illinois Board ofHigher Education, regional 
accreditation through the Higher Learning Commission, and specialized accreditation achieved through nationally recognized 
agencies and institutions. GSU seeks to capitalize on this alignment and has developed a five-year cycle for program review that 
enables programs to utilize annual assessment findings and self-study documents for multiple purposes. The emphasis has shifted 
from how much assessment data can be amassed to how the data can be used most effectively. This review process has also been 
designed to empower programs that do not have specialized accreditation. As part of the new review process, programs in the liberal 
arts and sciences and professional programs are required to complete self-studies that include external peer review. To facilitate this 
process, a template, which mirrors the self-study processes completed by programs that have specia 1 accred itation, has been created. 
Outcomes assessment is the base upon which the template and the redesigned program review process are built. Programs that 
complete the process are not only validated through this newquality assurance, but they are also better positioned to acquire needed 
resources. Moreover, the new administration welcomes this “burden.” 

In the past two years, all of these processes have begun to work in concert with one another and have contributed significantly to the 
emergence of a culture of assessment at GSU. A final factor that has ensured the emergence of this culture is the recognition that 
people make assessment successful, and GSU is fortunate to have many hard-working faculty members and administrators who 
willingly serve on multiple committees to ensure the success of our collective assessment efforts. The stability of university 
assessment committee, academic program review committee, and university curriculum committee (to name only a few) has been 
remarkable in recent years. Moreover, the fact that the reporting lines for these committees have blurred may serve as the university’s 
best evidence for its culture of assessment. For example, few on campus are aware that the university assessment committee is a 
provostial committee and not a faculty senate committee, as are the others listed above. This difference is unapparent largely because 
all of the aforementioned committees report at the monthly meetings of the faculty senate, and all share the same administrative 
liaison-the director of the Center for Quality. Such seamless integration has not only contributed significantly to the success of the 
individual committees, but it has also strengthened the overall culture of assessment on campus. 



Future Issues and Opportunities 

As at many public institutions throughout the country, budget cutbacks at GSU may create an uncertain future for assessment 
resources (e.g., travel to conferences, teleconferences, subscriptions). Internal funds may need to be reallocated, and competition for 
remaining resources may become intense. Depending upon the extent of the rescissions that may be necessary, large expenses such 
as the assessment software license may be reconsidered. In this impending era of fiscal austerity, there are those who predict, “No 
more teleconferences...just more (recycled) brown bags!” 

Whatever the immediate challenge, the long-term goal for GSU is to keep the assessment conversation going through day-to-day 
assessment activities and by means of assessment and accreditation scholarship. Our faculty and administrators are strivingtomake 
GSU a recognized leader in assessment and accreditation processes. We pursue this lofty goal because we genuinely recognize the 
value of assessment. After a decade of assessment turmoil, in just two years the university has witnessed some of the many benefits 
of a coherent and consistent assessment initiative. In the long run, it is hoped that academic and support unit validation through 
assessment will boost enrollment, heighten alumni pride, and better enable GSU to remain attentive to the needs and interests of its 
working adult students who stand at the core of its mission. Quality issues are particularly important to this population, and the 
university feels an obligation to deliver the highest quality programs possible. 
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Demonstrable excellence and continuous improvement made possible through the assessment of student learning outcomes must 
survive and flourish in even the most difficult of times. At Governors State University, we believe that this tenet will not only help us 
cultivate a culture of assessment today, but will also guide that culture well into the future. 

■ ■ 3.7 
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Cultivating a Culture of 
Assessment of Student Learning at 
Rochester Community and 
Technical College 

Tammy J. Lee and Anne M. Niccolai 



Rochester Community and Technical College (RCTC) was created in 1915 and is the oldest community college in Minnesota. The college 
offers seventy technical and transfer programs and serves nearly 6,500 students in credit-based programs and 11,000 in workforce 
education offerings. Located at the University Center Rochester (UCR), RCTC collaborates with Winona State University and the 
University of Minnesota to offer learning opportunities from certificate to doctoral programs. 

To guide the future design of RCTC, an innovative approach linking planning, quality, and accreditation was implemented. RCTC 
initiated a formal strategic planning process called Innovative Designs in March 1999 with the intent to connect strategic planningand 
the master academic plan with a Baldrige quality improvement process. 

Three of the steps in the strategic planning and implementation process were 

1. Realignment of the college mission and vision to position the college for the twenty-first century 

2. Identification of design criteria and academic performance indicators to determine points of reference for assessing quality 
performance 

3. Implementation of a comprehensive assessment of student learning initiative 

Realignment of Mission and Vision 

A major step in realignment included a review of college mission and the development of a vision statement that would position RCTC 
for the twenty-first century. Once established, the mission, vision, and belief statements were later used to support the identification 
of key performance indicators for academic program assessment. Only one change, the addition of “quality,” was made to RCTC*s 
current mission. It was added to communicate the college’s commitment to continuous quality improvement. 



RCTC Mission 

Rochester Community and Technical College provides 
accessible, affordable quality educational opportunities to a diverse community. 



Creating a vision that would position RCTC for the twenty-first century was the next stage in the plan ning process. RCTC had operated 
without a vision statement since the merger of Rochester Community College and Minnesota Riverland Technical College in 1996. 
Usingthe findings from an environmental scan, a taskforce developed a draft vision statement: “Rochester Community and Technical 
College will be a world class provider of liberal arts, technical, and life-long learning.” This vision statement was communicated to all 
staff at a September 2000 staff development day. Faculty and staff were asked to define (in writing or pictures) what this vision 
statement looked like to them when achieved. Their involvement created a better understandingofthe issues and challenges involved 
in a quality improvement process. It also enabled them to begin building a continuous improvement culture. 



Identification of Design Criteria and Academic Performance Indicators 

Another step in the planning phase involved the establishment of design criteria. The design criteria (aims or goals) are features that 
must be present within academic programs and departments in orderto be responsive to the challenges and issues facing RCTC. They 
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served as the basis for the academic division analysis. The task force identified over a hundred design criteria. This list was later 
narrowed to eighteen through implementation of a nominal group brainstorming process. Faculty and staff further narrowed these 
criteria to three using a 10-4 quality-voting tool. Faculty and staff were asked to identify from the list of 18, those design criteria that 
“best” align with the future vision of RCTC. Using ten stickers, faculty could place a maximum of four stickers on any one criterion. The 
three RCTC design criteria resulting from implementation of quality processes and tools are as follows. 

1. The mission and vision are modeled in the delivery and support of teaching and learning. 

2. The college aligns resources to support the mission and vision. 

3. The performance of the college demonstrates continuous Improvement. 

Upon development of the three design criteria, the college faculty began to identify key academic performance indicators to monitor 
their academic program's overall performance. To establish these performance indicators for each design criterion, rubrics were 
created. These key performance indicators provided faculty with a standard for Identifying strengths and opportunities for improve- 
ment within their programs. 

□ Performance Indicator Framework 

A rubric framework was developed to establish a common language around RCTCs three design criteria. Development of the key 
performance rubric framework involved RCTC's program and department chairs, faculty, and academic deans. Involvement of 
faculty enabled the faculty to direct their own performance process more consistently and effectively. It also created a better 
understanding of the Issues and challenges involved in an assessment process. 

Two staff development sessions in designing effective rubrics were conducted. The first session, “a chocolate-tasting session 
model,” provided general guidelines for rubric development and helped faculty understand rubric design. Faculty members were 
asked to develop a matrlxor rubric that identified key performance indicators ofaquality chocolate. Afour-level rubric Identifying 
characteristics at the exemplary, acceptable, improving, and not acceptable levels was developed. Faculty then tested their 
rubric assessment using various quality chocolate candies. 

A second staff development session focused on identifying specific performance indicators and rating levels for each of the three 
design criteria. Once established, the rubric framework served in the process of self-examination by the college faculty to 
determine if their academic program aligned with the mission and design criteria of the college. 



Implementation of Comprehensive Assessment of Student Learning Initiative 

Following our accreditation site visit in spring 2002, It became apparent the college needed to develop a comprehensive assessment 
of student learning model and begin the implementation process as soon as possible. As an Initial step toward that goal, a team of 
four attended the American Association of Higher Education (AAHE) Assessment Workshop. The workshop allowed the team the time 
to examine the college mission, vision, design criteria, and academic goals, and build a comprehensive assessment of student 
learning plan. The workshop was structured to facilitate an assessment plan that Included goals, action steps, measures of 
attainment, and budget. Each team was assigned a mentor who guided them through the process, interactive learning sessions, best 
practice models, and work sessions. Focused time away from campus with wonderful resources allowed team members to develop 
a working document for future use. Draft plans were shared with all workshop attendees during the final workshop session. 

As a college, workshop attendance put us on the road to implementation. Once back on campus, the team further defined the plan 
and added the budgetary component. Adding a budget allowed the college administration to begin to understand the commitment 
and focus that would be necessary to Implement a comprehensive assessment of student learning plan. It also helped the college 
understand how assessment of student learning is related to institutional effectiveness. 

Once a comprehensive assessment of student learning plan had been approved by college leadership and faculty, an evaluation of 
our current status was necessary. The college decided to adapt the levels of assessment developed by Cecilia Lopez (Higher Learning 
Commission) to reflect RCTC's profile. Focus groups representing leadership, faculty, and students scored current level placement. 
The long-range plan includes annual evaluation of the levels of assessment to document future movement among the levels. This 
activity focuses future planning and development activities toward moving the college to level three (implementation). 

An initial professional development forum included presentations by two colleges similar to RCTC that had mature assessment of 
student learning initiatives. One of the colleges utilized standardized tests to assess student learning, while the other used locally 
developed assessment measures. This experience allowed faculty and leadership the opportunity to evaluate two models and begin 
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to plan implementation activities within our institution. Both best practice colleges provided invaluable resources for future 
development on our campus. 

Following the best practice showcase, a comprehensive assessment of student learning committee with faculty representation from 
multiple disciplines and leadership was cultivated. One of the primary goals of this committee was support of pilot projects within 
several general education disciplines. Committee involvement from general education faculty and leadership was critical to the 
success of the pilot projects. The thoughtful design of this committee allowed the college to gain the commitment of key faculty and 
academic leadership. This step was essential in the cultivation of a culture of assessment at RCTC. 

During this time, the college had the opportunity to take part in a project being carried out by the system office at the state level. A 
data-tracking software program had been developed and was seeking pilot participants. Two college programs that were currently 
assessingstudent learning participated in pilotingthe software. The pilot project resulted in further development ofthe data-tracking 
software to meet the needs of occupational programs as well as general education disciplines. 

A major focus of our 2002-2003 assessment of student learning plan was to give faculty time to explore assessment models and 
integrate successful practices into a general education pilot. To explore national best practice programs, six faculty members and one 
administrator attended the AAHE Assessment Conference in June 2002. The faculty members were assigned to complete a summary 
discussion worksheet intended to (1) list two examples from the conference that were effective in measuring general education 
competencies, (2) identify presenter(s) who could be considered for campus staff development days, and (3) brainstorm a list of theme 
ideas that would help communicate our assessment initiatives to faculty and staff. When asked to evaluate the AAHE national 
conference, the faculty members reported that they had a greater understanding ofthe depth of assessment across the country and 
their role in the multiple levels of assessment. 

Following the AAHE Assessment Conference, summer pilot planning sessions were held over a three-day period. This provided faculty 
with time to reflect on what they learned at the conference and embed concepts into a general education assessment pilot for RCTC. 
Each discipline was given a planning template and asked to select one Minnesota Transfer Theme area to pilot, identify who would 
be assessed, determine the available data, identify additional data needed, identify the student learning assessment criteria and 
assessment tool to measure learning, and determine ways to use the data. Pilots were developed for the English, math, and science 
disciplines. 

Building an understanding and awareness of our student learning assessment initiatives was key to achieving our goals. At our fall 
staff development day, information on the pilots was shared with all faculty and staff, followed by a stinger cafe session that engaged 
all programs and departments in identifying one student learning outcome that they would assess during fall semester. In addition, 
each member of the assessment of student learning committee was asked to work as a department/program in developing 
communication messages for one assigned month. The member responsibilities included 

o Posting one student learning assessment highlight from their department/program on the RCTC assessment Web site 
<acd@roch.edu/asl> 

o Publishing a brief summary of the student learning assessment highlight in the staff newsletter 
o Writing a brief summary of an assessment book, video, or conference, and posting the summary on the RCTC assessment Web site 

o Recording the minutes for one committee meeting and posting them on the Web site 

Ourfuture plans include assessing, analyzing, and adjustingourgeneral education model based on thefall pilot results. AMarch 2003 
staff development day is scheduled to bring in Peggy Maki, Director ofthe AAHE, to work with faculty to expand their work. We also 
plan to expand into the program areas and begin implementing a similar process that we used in developing our general education 
assessments. A team of program faculty will have opportunities to explore best practice models and summer planningtime to develop 
pilots within their own programs. 

While we have made progress on the road to a fully implemented student learning assessment program, much ofthe journey lies 
ahead. Our main goal is to keep faculty engaged and motivated while minimizing barriers to implementation. Feedback relating to our 
process has been positive. Additional faculty are steppingforward with fresh in novative ideas relatingto assessment. We lookforward 
to continued progress at our institution. 



Gardening Tools Used for Cultivation 

o Cafe-Style discussion groups consist of small groups (four to six people) discussing a topic for a set amount of time and then 

moving to another group. Each table has an assigned person who stays at the location and records the essence of the 
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discussions. This style of discussion keeps dialogue moving, mixes up participants, and keeps one or two people from 
dominating the discussion for the entire time. 

o The affinity diagram consists of ordering items through a nominal ranking system, and 10-4 voting gives each person the 
opportunityto assign ten votes using a maximum of fourvotes forone item. The following example illustrates how it was used 
in the development process of our college design criteria. People were given ten stickers and asked to vote for the design 
criterion that most reflected our college mission. They could place a maximum of four stickers next to one design criterion, 
and they had ten votes to assign. This activity was repeated for the design criterion that least reflected the mission. Following 
the 10-4 voting activity, an affinity diagram was constructed using the results. We then had a nominal ranking of our design 
criteria. 

o Rubric development was incorporated into a professional development activity. A rubric development framework that could 
be adapted to a variety of situations was presented to faculty. Following the presentation, a group activity resuited in the 
development of a four-level rubric identifying characteristics at the exemplary, acceptable, improving, and not acceptable 
levels for chocolate. Faculty then tested their rubric to assess various qualities of chocolate and refined the criteria. 

o The levels of assessment were deveioped by Cecilia Lopez oftheFligher Learning Commission. Colleges use them to chart their 

movement toward a fully implemented assessment of student learning initiative. The wording in the original document is 
genericso as to adapttoanyinstitutionof higherlearning. We adapted thedocumentto reflect intent forRCTCasacommunity 
and technical college focusing on occupational programs and transfer education. The adaptation included the use of terms 
consistent with other college documents and contained in the glossary of terms developed by the college. 



The pilot process implemented by the RCTC assessment of student learning committee focused on a quaiity plan-do-check-act model. 
Faculty members were given time to (1) explore national best practice models, (2) develop a plan to integrate successful strategies into 
a piiot student learningassessmentfortheirdisciplines, (3) implementthe pilot for one semester, (4) analyze the resuitsoftheirpilots, 
and (5) make adjustments before implementing a campus-wide system. 



Tammy j. Lee is Business instructor and Co-Assessment Coordinator at Rochester Community and Technical College in Rochester, 
Minnesota. 

Anne M. Niccolai is Director of Dental Hygiene and Co-Assessment Coordinator at Rochester Community and Technical College in 
Rochester, Minnesota. 
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Assessment of Student Learning: 
Creating Change That Lasts 



College Profile 

Shepherd College is part of the public higher education system of West Virginia. Located in the historic eastern panhandle of West 
Virginia and perched on the Potomac River, the school attracts students primarily from West Virginia, Virginia, Maryland, Washington, 
D.C., and Pennsylvania. Approximately fourthousand students are enrolled ini97 undergraduate majors, minors, and concentrations; 
the college also has a community college component and is currently developing selected graduate programs. 



Assessment as Change Agent 

In 1998, Shepherd College found itself in a predicament that many colleges and universities can relate to-slated for an accreditation 
visitin2oo2,thecampusassessmentefforts had been stalled at everyturn.Anewdirectorwas assigned to lead the assessmentefforts 
on campus, and a “formula” for assessment plans and reports was established. Over the last four years, however, the “formula” was 
transformed into a “vision,” and the result was a campus*wide assessment program that effected dramatic changes in attitudes about 
assessment. 

Harvard Business School professor John Kotter outlines an eight-stage process of creating major change in his book Leading Change 
(Harvard Business School Press, 1996). The eight-step process that Kotter identifies interfaces with the assessment process 
implemented on Shepherd's campus; this theoretical framework of effective change helps explain why the assessment process was 
successful and will provide a framework for other institutions establishing an assessment process. 



(excerpted from Leading Change, 1996, Harvard Business School Press) 

Step 1. Establishing a Sense of Urgency 

o Examining the market and competitive realities 
o Identifying and discussing crises, potential crises, or major opportunities 

Step 2; Creating a Guiding Coalition 

o Putting together a small group with enough power to lead the change 
o Getting the group to work together as a team 

Step 3: Developing a Vision and a Strategy 

o Creating a vision to help direct the change effort 
o Developing strategies for achieving that vision 

Step 4: Communicating the Change Vision 

o Using every vehicle possible to constantly communicate the change vision 
o Having the guiding coalition model the behavior expected of employees 



Patricia Dwyer 



John Kotter’s Eight-Stage Process of Creating Major Change 
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Step 5: 


Empowering Broad-Based Action 




0 


Getting rid of obstacles 




0 


Changing systems or structures that undermine the change vision 




0 


Encouraging risk taking and non-traditional ideas, activities, and actions 


Step 6: 


Generating Short-Term Wins 




0 


Planning for visible improvements in performance, or “wins” 




0 


Creating those wins 




0 


Visibly recognizing and rewarding people who made the wins possible 


Step/: 


Consolidating Gains and Producing More Change 




0 


Using increased credibility to change all systems, structures, and policies 
that don’t fit together and don’t fit vision 




0 


Hiring, promoting, and developing people who can implement the change 
vision 




0 


Reinvigorating the process with new projects, themes, and change agents 


Step 8: 


Anchoring New Approaches in the Culture 




0 


Creating better performance through customer-oriented and productivity- 
oriented behavior 




0 


Articulating the connections between new behaviors and organizational 






success 




0 


Developing means to ensure leadership development and succession 



Kotter’s Eight Stages and Their interface with Shepherd College’s Assessment Process 

Step 1. Establishing a Sense of Urgency 

o Examining the market and competitive realities 
o Identifying and discussing crises, potential crises, or major opportunities 

Facing an accreditation visit was all the urgency we at Shepherd College needed. But the “major opportunities” cited in this 
stage can provide a more positive spin to the “urgency” thatgets a campus to move forward with an assessment process. Some 
opportunities may include major curricular reform, a realignment of schools or programs, or outside funding for a particular 
project that demands assessment. 

Step 2: Creating a Guiding Coalition 

o Putting together a small group with enough power to lead the change 
o Getting the group to work together like a team 

Shepherd College had had a plethora of assessment committees but no action. When putti ng together a guiding coalition, the 
director invited selected faculty and staff members who would work well together and would be good ambassadors for the 
assessment program. In addition, the name change to “Assessment Task Force” contributed to the different tone that the 
group hoped to communicate. Unlike other committees that tend to be top-heavy with administrators, the task force had one 
dean who represented the other administrators. Staff, faculty, and— most importantly— students were invited to participate. 

The taskforce met once a month and planned assessmentworkshops, helped departments construct assessment plans and 
reports, and generally offered assistance where needed. 

Step 3: Developing a Vision and a Strategy 

o Creating a vision to help direct the change effort AVAIMBILIE 

o Developing strategies for achieving that vision 
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This step was perhaps the most important in the assessment process at Shepherd College. Our vision took the form of 
connecting assessment to improving student learning. That was the mantra whenever the director was discussing assess- 
ment issues with department chairs, faculty, staff, or students. The assessment office became “Assessment of Student 
Learning”-the administrative assistant answered the phone with that phrase, the office stationery communicated that 
message, and every newsletter and announcement always linked assessment and student learning. At the same time, the 
director proposed adding a statement to the institutional mission that would highlight the importance of improving student 
learning. The following was added to the college mission statement in 2000: “Student learning is central tothe culture of ours 
and finding ways to improve student learning is a continuous process.” 

Strategies for achieving the vision included yearly assessment plans and reports from each department and unit. While this 
cycle demanded quick turnaround to complete the assessment loop, it also provided more immediate feedback (and 
deadlines) for departments that were starting assessment activities for the first time. In addition, departments identified 
three-andonlythree— learninggoalsto beassessed eachyear. This keptthe process from seemingoverwhelming, especially 
for beginners. 

To support assessment efforts, the director invited each department to go off campus for an assessment retreat. The Office 
for Assessment of Student Learning made arrangements and paid for meals, and the director offered to facilitate a discussion 
with the department about program learning goals. Throughout the academic year, the director organized grassroots faculty 
development opportunities to continue educating the campus about assessment. 

Step 4: Communicating the Change Vision 

o Using every vehicle possible to constantly communicate the change vision 
o Having the guiding coalition model the behavior expected of employees 

The Office for Assessment of Student Learning communicated the vision in three key ways. A newsletter, “Assessment of 
Student Learning at Shepherd,” showcased assessment’s link to student learning. “Focus on Student Learning” featured 
faculty and staff as guest speakers who addressed topics of campus-wide interest, and a brown-bag lunch group, “Food for 
Thought,” discussed issues of teaching and learning. 

Theguidingcoalition,ourAssessmentTaskForce,servedassupportforthecampus,andincommunicatingwith departments 
or individuals always linked assessment with student learning. The task force members reviewed yearly plans and gave 
feedback and advice about format. Thus, they made sure that each plan was linked to the mission, had three learning 
outcomes to be assessed, had two means of assessing each outcome (both direct and indirect), and had benchmarks to 
determine success. Task force members did not comment on the actual outcomes being assessed but gave feedback only on 
the components of a good plan. 

Step 5: Empowering Broad-Based Action 

o Getting rid of obstacles 

o Changing systems or structures that undermine the change vision 
o Encouraging risk taking and non-traditional ideas, activities, and actions 

Getting rid ofobstacles? Perhaps a more accurate description on ourcampus was working around the obstacles. For example, 
the director called her team a task force rather than a committee. This kept the group from stepping on the toes of already 
established assessment committees that didn’t want to move the process forward. In addition, the task force was housed 
under strategic planning rather than the faculty senate, known for its propensity to stall or obfuscate important issues. 

Early on in the assessment process, the task force wanted to encourage departments to assess those outcomes that would 
be more risky; in other words, departments should not assess only those outcomes that felt safe or were guaranteed to 
produce good numbers. Thus, the task force created a campus “philosophy of assessment” in which they asserted that 
assessment results would not be used for punitive purposes. 

Initially, each department was automatically awarded $500 to defray the cost of assessment materials. At the end of the first 
year under the new director, very little of the money had been utilized; most departments did not have ideas, other than 
ordering tests, of how to use the funds. In the second year, the director shifted to mini-grant applications and encouraged 
departments to apply for projects. After the first year of awards, projects were advertised to the general campus population 
as a way to generate more ideas for using assessment funds. 

Step 6: Generating Short-Term Wins 

o Planning for visible improvements in performance, or “wins” 




45 



3 S //1 Collection of Papers on Self-Study and Institutional Improvement, 2003 - volumes: Promoting Student Learning and Effective Teaching 



o Creating those wins 

o Visibly recognizing and rewarding people who made the wins possible 

Our wins in the first year were simply getting assessment plans and reports completed for each department/program. While 
they weren't perfect, the completion of the assessment loop gave departments some data to work with, and department 
members discovered that they could learn something valuable about their programs. As assessment cycles afforded more 
information to the departments, they were encouraged to use results in seeking budget increases. 

We visibly recognized these wins by showcasing departments' assessment efforts in the newsletter or through the speaker 
series. In addition, we started a student achievement day; here students submit proposals to present projects or papers to 
the school community. 

Step 7: Consolidating Gains and Producing More Change 

o Using increased credibility to change all systems, structures, and policies that don't fit together and don’t fit vision 
o Hiring, promoting, and developing people who can implement the change vision 
o Reinvigorating the process with new projects, themes, and change agents 

In 2002, with four years of assessment data collected, our work was rewarded in several ways. We received a commendation 
on ourassessment efforts from our HigherLea rningCommission accreditation team. Thecollegealso restructured and created 
the position dean of teaching, learning, and instructional resources to expand on assessment activities and to develop a 
centerforteachingand learning. Newactivitiesincluded studentandfacultylearningcommunitiesand a reinvigorated writing 
across the curriculum program. 

Step 8: Anchoring New Approaches in the Culture 

o Creating better performance through customer-oriented and productivity-oriented behavior 
o Articulating the connections between new behaviors and organizational success 
o Developing means to ensure leadership development and succession 

Our better performance included changes in the curriculum, prompted by assessment, to better meet student needs. 
Organizational success in our accreditation visit contributed to making assessment more a part of the fabric of the institution. 
Newleadership has emerged through thefaculty learning communities, thein-housespeakerseries, and the interdisciplinary 
student learning communities that bring faculty together in teaching initiatives. A summer institute is planned for faculty to 
revise courses based on active learning strategies. 



References 
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Taking Assessment Off the Shelf: 
Developing a Public 
Culture of Assessment 

Sandra S. Bowles and Alan R. Belcher 

Overview 

In the 1990s colleges across the country developed assessment plans to meet the requirements of accrediting agencies. Too often, 
when the visiting team left campus, the assessment plan was shelved. This case study examines how one institution has moved the 
total institution toward full implementation of its assessment plan and has made significant progress toward attaining Level III of the 
Assessment Matrix in two short years. 



Background 

In 1993-1994 the vice president for academic life at the University of Charleston appointed an ad hoc assessment committee charged 
with developing an assessment plan for the institution in preparation for a 1994-1995 NCA visit. The committee worked diligently to 
inventory assessment activities over the campus and prepared a plan that stood the test of approval from the visiting team and the 
Commission. Everyone breathed a sigh of relief, and the assessment plan was filed in the library and on committee shelves, where 
it did little more than collect dust. This scenario is probably not too unlike what has happened on many college campuses. 



What Changed? 

In fall 1995, the faculty passed the following motion: “The faculty commits to developing an educational system which encompasses 
the concepts ofanoutcomes-based program with comprehensive and continuous assessment of learning.”Thiswas followed with the 
creation of faculty roundtable groups charged with developing a set of liberal learning outcomes that would replace the general 
education requirements used by the institution for many, many years. Faculty members worked diligently to sift competencies in six 
categories-citizenship, communication, creativity, critical thinking, ethical practice, and science. For each category the roundtables 
identified competencies that would be required of each baccalaureate graduate of the University of Charleston. 

Liberal learning outcomes (LLOs) are defined as the knowledge and skills one needs to live a productive and meaningful life in the 
twenty-first century. The LLOs program at the University of Charleston is designed to provide experiences that enable students to 
develop skills, adopt values, and acquire knowledge that will promote their achievement of worthy personal and social goals in the 
context of a rapidly changing society. LLOs are not limited to the presentation of factual knowledge. Students must be able to control 
knowledge: to produce, organize, communicate, and use it. Writing and problem solving (including analysis, synthesis, analogy, and 
metaphor) are major elements of the program. A liberal education is one that enables students to evaluate the power and limitations 
of different intellectual, aesthetic, moral, and religious methods. This ability allows students to develop those personal qualities that 
promote productivity and satisfaction. Learning experiences at the university foster the development of the student's abilities to 
demonstrate mastery of the LLOs required of successful graduates. The experiences also reflect the university's mission, which is to 
“educate each student for a life of productive work, enlightened living and community involvement.” 

At the same time, the first-year curriculum underwent significant change. Content from courses traditionally required of freshmen was 
reorganized into three learning communities-science, social science, and humanities. First-year students also engage in an 
orientation to university and university experience and a residence-based living-learning community. Within these communities the 
foundational competencies associated with the liberal learning outcomes are integrated with content and knowledge competencies 
in science, social science, and humanities. These learning communities, the skill-building workshops that support the learning 
communities, and the mentoring program all comprise the initial college experience (ICE) program for incoming freshmen. Students 
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earn academic credit when they demonstrate that they have learned the skill and/or knowledge associated with the outcome. Student 
mastery of the competencies is evidenced through a portfolio that can be evaluated at any time during the student's progression 
through the academic program. 

Beyond the freshman year, the liberal learning outcomes are embedded within the content and evaluation of discipline-specific 
coursesorcourses from an elective or related field. Coursesthat have achieved roundtableapprovalfordeliveryofLLOsaredesignated 
as ICON or portfolio-bearing courses. Approved courses appear in the academic course schedule with an indicator (ICON) that 
indicates which outcome(s) the course can help students achieve. Students must demonstrate achievement of the specified LLOs for 
successful completion of the course. As an alternative to achievement of specific liberal learning outcomes, students may complete 
a series of portfolio-bearing courses and present a collection of their learning products to demonstrate achievement of an outcome. 
Portfolios are submitted periodically for assessment and feedback. A senior capstone course currently under development may offer 
opportunity for final fulfillment of LLOs. Assessment of student achievement is based on criteria established by the individual 
roundtables and made available to faculty. 

To further complete the curricular requirements at the university, over the academic year 2001-2002 each program identified 
measurable learning outcomes that each student enrolled in the program must achieve to graduate. 

Students at the university have the opportunity to move at their individual level of ability. Students are encouraged to work at their 
own pace and to avail themselves of the opportunity for “learning your way,” which is the university's way of communicating that 
students may complete programs in less than four years or may extend the academic experience as needed. 



Implications for Faculty 

As one considers the several major changes that have been made in the organization and delivery of the curricula in this institution, 
it is readily apparent that the traditional approaches to course syllabi development, teaching, and assessment required a great deal 
of modification. To clearly identify course outcomes and appropriate assessment, it became necessary for faculty to redesign course 
syllabi, to clearly identify how each competency is assessed, and to identify the minimal standard required for the student to 
demonstrate achievement of the competency. It was painfully apparent that faculty needed to rethink thei r approach to teaching and 
the determination of student success. The whole process mandated that faculty become more knowledgeable and skilled in 
assessment of student learning. For the institution, this required a huge commitment to faculty development. 



Raising Assessment Awareness 

Engaging the institution and particularly the faculty in effective assessment has become a way of life for the University of Charleston. 
Through a variety of activities and initiatives, we have successfully raised institutional awareness of the value and processes of 
assessment. Throughout the process we have repeatedly stressed the need to keep it simple and to encourage university constitu- 
enciesto collect data only ifthey can be used to improveteachingand learningor help the universityachieve its mission. This emphasis 
has served us well in working with faculty, as it has helped take much of the mystique out of assessment as a basic essential to the 
buy-in and use of assessment principles. Overtime we have managed to engage all constituencies of the university, albeit some more 
successfully than others. We have made great strides in promoting assessment as a part of the institutional culture. 



What Did We Do? 

I. Promoted faculty buy-in 

Prior to 2001 our faculty had limited expertise in working in an outcome-based curriculum and the requisite use of assessment. 
Faculty were somewhat resistant to change and encumbered with heavy workloads. Priortothis time there was no one in charge 
of assessment. A number of activities helped us work through these problems, including 

A. Workshops 

1. Using the Higher Learning Commission’s “Levels of Implementation” as a framework for development of assessment 
across the institution 

2. Developing outcome-based programs 

3. Developing outcome-based courses 
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B. Use of consultants to assist faculty to develop assessment skills 

1. Dr. Virginia Anderson presented multiple workshops and consultation on assessment and development of assessment 
tools 

2. Dr. Charles Bonwell worked with faculty in teaching and assessing critical thinking. 

C. Faculty encouraged to give and take 

1. Presented in-house workshops 

2. Presentations for faculty development 

3. Faculty retreats to work on curricular development and assessment 

D. Attendance at national conferences and workshops 

1. AAHE attendance and presentations with groups of six to eight faculty members 

2. Attendance at Higher Learning Commission Annual Meetings 

3. AAC&U Institute on Sustainable Innovations 

4. Team often faculty members to Alverno Institute 

E. Institution of merit-based pay to reward faculty memberswho have actively engaged in curricular revision and implementation 
of effective assessment 

F. Collaboration with other schools 

1. Member of Appalachian College Association (ACA) sharing expertise and resources 

2. West Virginia Independent College Enterprise 

G. Regular periodic review of program strength and viability 

1. Academic planning committee reviews academic programs every two years 

2. Makes recommendations for program enhancement or closure 

II. Added an assessment committee to the standing committees of the university 

Prior to 2001 there was little or no leadership for assessment on the campus. The appointment of a coordinator of faculty 
development, pa it ofthe duties ofthe director of learning, assessment and technology, was a major step forward. This, in concert 
with the addition of an across-the-campus assessment committee, has been very effective. 

A. Representation includes members from each of the liberal learning outcome roundtables, dean of student life, librarian, 
director of institutional research, the provost, students elected from SGA, and the director learning, assessment and 
technology (twelve members) 

B. Committee oversees assessment and promotes development of “culture of assessment” 

C. In 2001-2002 served as mentors to individual academic programs as each program developed program mission statements, 
outcomes, and appropriate assessment for each outcome. In fall 2002 these were evaluated by the committee, with feedback 
provided to each program. 

D. Assessment committee’s work made public 

1. Monthly reports to university faculty, which include sharing of results from assessment instruments and surveys 
administered to students and faculty 

2. Minutes online 

3. Statement of philosophy of assessment in catalog 

4. Currently developing Web-page 

5. Will coordinate spring 2003 assessment showcase 

6. Information shared with board of trustees through the academic life committee 
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III. Obtained funding to support faculty development 

A supportive administration actively sought and obtained funding to provide necessary resources for faculty development and 
curricular revision. This has been an essential component of our success. The involvement of faculty in local and national 
workshops todevelop expertise in curriculum and assessment could not have been accomplished withoutthe availability ofgrant 
funding. 

A. Major grants obtained from the Claude Worthington Benedum Foundation, Christian A. johnson Endeavor Foundation, and 
the Independent College Enterprise 

B. Grants used to provide a variety of initiatives, including 

1. Faculty retreat 

2. Workshop and conference attendance 

3. Consultants 

4. Funding for faculty mini-grants for curriculum development and the development of embedded and authentic assess- 
ment tools 

5. Faculty participating in in-service provided stipends 

Results/Benefits 

While we have moved our campus to an outcome-based curriculum and made huge strides in developing and implementing our 
assessment plan, there have been other, serendipitous outcomes that have had what we considerto be positive outcomes, including 

o The development of clearly stated outcomes for programs and individual courses, and outcomes that are connected to 
pedagogy and assessment 

o Faculty focus has changed from teaching to learning and demonstrates an increased emphasis on personal excellence in 
teaching 

o Faculty clearly see need to evaluate student achievement to support “learning your way” 

o Proficiency in developing assessment techniques and assessing student academic achievement has been instrumental in 
improving program effectiveness 

o Work ofthe assessment committee has resulted in greater interdisciplinary collaboration and improved working relationship 
among faculty members 

o The linking of liberal learning outcomes and professional education provides a more “seamless” education 
o Faculty members are excited about their work and willing to share 



Remaining Challenges 

While we have made significant progress in our efforts in curriculum development and assessment, we recognize that many ongoing 
challenges remain. Some of these include 

o Obtaining additional funding for continued faculty development 

o Orientation of new and adjunct faculty to outcomes-based curriculum and the importance of assessment 
o Student appreciation ofthe assessment process and benefits offered them 

o Continued development of authentic course-embedded assessment tools specific to our liberal learning outcomes 
o Research in the use of assessment to improve teaching and learning 
o Assimilation of transfer students into an outcome-based curriculum 
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Conclusion 

The University ofCharleston has moved toan outcome-based curriculum that demands assessmentofstudent learning. We have made 
significant strides in engaging administration, faculty, and students in assessment. Focusing on faculty development and internally 
sharing expertise, we have successfully engaged faculty in assessment at levels far exceeding our expectations. 

Change is never easy, and we have sometimes lost sight ofwhat we are tryingto accomplish because we have at times become bogged 
down in the details. However, faculty members at the University ofCharleston are learning to challenge themselves and each other. 
They are learning to use assessment effectively, to find alternative ways of accomplishing goals, and to be certain that we are being 
true to our students and our mission. Even with the challenges of meeting internal and external demands on our time and energy, we 
are committed to continuing to evolve, purposefully and strategically, into our dream of a learner-centered institution. To this end we 
have encountered many challenges, but we have made significant progress toward taking assessment off the shelf, and we are willing 
to share our experience. 
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Integrating Assessment 
into College Life: A Faculty- Driven 
Checklist of Strategies 



Barbara Edwards, Vincent Holtmann, Kelley Raftery, 
and Brad Sparks 



Introduction 

How does outcomes assessment become integrated into the culture of an institution? The answer, it seems, is mostly by trial and 
error-and very slowly. A faculty member may have an idea that works with his or her students and may pass it on to a colleague. An 
administrator may bring back a suggestion from another institution, or an alumni membermayattend an advisory board meetingand 
mention something that helped him or her succeed in the workplace. Over time techniques and ideas evolve, change, and are 
sometimes discarded, but some strategies are successful in fosteringan assessment climate at the institution. This paper will review 
thetechniques that have gradually caused a culture shift at Ranken Technical College, changingit from an institution that reluctantly 
complied with the Higher Learning Commission’s outcomes assessment initiative to one that recognizes the significance of, and 
actively promotes, a continuous improvement process in student academic achievement. 



Institutional Background 

Ranken Technical College (RTC) is a private, non-profit institution of higher education that awards associate degrees in thirteen 
technical fields and a bachelor of science degree in architectural technology. Located in St. Louis, Missouri, it was founded in 1907 

by David Ranken, Jr., and as a result ofthe ever-increasing complexity oftechnology and greater demands in the workplace, the college 

hascontinued to expand its curriculum and broaden its programs. Ranken offers a unique three-part educational formula that blends 
general education classes and technical education in theory and laboratory courses with a dedication todevelopingpositive work ethic 
standards in students. In this role the college has always maintained close ties with the industries that hire Ranken graduates, and, 
as the college approaches its hundred-year anniversary, this philosophy continues to be a factor in the success of its graduates. 



Outcomes Assessment History 

lni999,afteracomprehensiveinstitutionalvisitfromtheHigherLearningCommissioninwhich the institution wasgranted continued 
accreditation with the next comprehensive evaluation in ten years, Ranken was challenged with a monitoring report on outcomes 
assessment. At that time the Higher Learning Commission felt that while the college had a long history of gathering and evaluating 
data about institutional effectiveness, its outcomes assessment plan was still in an “embryonic” state. The assessment leadership 
at the college realized that, while considerable effort had been made to improve student academic achievement during the previous 
years, it was crucial for assessment to become part ofthe college culture. A major shift would have to take place that would involve 
everyone in the college-and beyond-in the assessment process to make it a priority. 

Until then, a small group of faculty had struggled with assessment issues with limited success. Departments had been required to 
write assessment plans without a real understanding ofthe underlying goal of improving student learning. Facultyviewed assessment 
as just one more task in their already busy schedules, and administrators viewed it as compliance rather than quality improvement. 
In an institution that prides itself on the work ethic and traditional values, the requirement of a monitoring report acted as a wake- 
up call indicating that the college was not performing at the highest possible standard. Those involved in assessment realized that 
improvingthe assessment program required buy-in from all constituents ofthe college. The challenge of a monitoring report acted as 
a catalyst to gain the necessary support that ultimately made this possible. 
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Over the next four years awareness of assessment expanded, and more people became actively involved In the process. Faculty 
became more willing to try new ideas. Strategies that proved successful fueled enthusiasm for further efforts. Those Involved in 
assessment realized that strategies that are ongoing and, once established, self-perpetuating were most successful in developingan 
environment for growth in assessment. They also realized that the complexity of assessment could be reduced with a meaningful 
assessment plan and viable strategies to achieve goals. A review of the strategies that have reinforced assessment at the college 
targets four specific groups: 

o Administration 

o Faculty 

o Students 

o External constituents 

The following discussion focuses on strategies that were developed by. and/or directed at, each group in turn. 

□ Administration 

While most strategies for assessment were introduced and sustained by faculty, after the 1999 evaluation the administration of 
the college initiated a number of ideas that greatly facilitated and encouraged this process of assessment for the faculty. Upon 
the advice of the Higher Learning Commission, the administration provided a central repository for the mounting outcomes 
assessment data. Additionally, clerical support in the form of a staff member to maintain records, take committee meeting 
minutes, assemble documents, and do filing was approved. One faculty member was given release time to compile data and 
rewrite the outcomes assessment (OA) plan in conjunction with the outcomes assessment committee. Also, after tracking 
expenses for a year, the committee was given its own line-item budget; and assessment became embedded in the planning 
process of the institution by beingtied into departmental budget requests. Proposed budget requests are directly linked to goals 
set by departments, which in turn affect student learning. These goals are reviewed annuallyand are now compared to outcomes 
assessment data. Additionally, outcomes assessment has been added to the annual faculty evaluation process. Reinforcement 
was also given to assessment efforts through the acknowledgement ofOA accomplishments on annual employee appreciation 
days. Finally, an outcomes assessment session was formally added to the three-day/7ew/ocu//yc/eve/opme;?f5em/>?arpresented 
semi-annually by the vice president for institutional advancement. Introducing new faculty to the role of assessment was felt to 
be an important step in furthering outcomes assessment at the college. 

□ Faculty 

The basis for Ranken's assessment program is a strong oufcomes assessment committee com\)ose(\ of faculty from all academic 
departments. However, prior to 1999, the committee was often less than successful in establishing a systematic program for 
measuring and Improving student academic achievement at the college. At an institution steeped in tradition, major reasons for 
the committee's lack of success Included faculty resistance to change and an increased workload. Administrative support has 
been partially responsible for increasingthe committee's effectiveness, as has the fact thatonce instructors saw first-hand how 
meaningful feedbackcould improve student learning, they gradually became more enthusiastic aboutthe assessment program. 
Meetings are held bi-weekly, and minutes are maintained in the central repository and on the college's intranet. Departments 
rotate members annually to give everyone a chance for direct participation. A number of faculty members feel that the increased 
communication between departments has been beneficial in determining how the curricula of related departments (for example 
control systems technology and industrial electricity/electronics technology) complement each other and/or fit together. 
Assessment leaders are also available to meet with departments individually to help with their individual assessment plans and 
with solutlonsto any problems. The committee generally sets annual goals and aims to provide a supportive environment for the 
practice of assessment, as well as a forum for assessment issues. 

All academic departments at the college hove developed individual outcomes assessment plans, along with a calendar of 
important assessment dates. Additionally, departments are asked to maintain all assessment data in a departmental activity 
binderthat is updated on a regular basis. Information in the binders varies by department and may include student critiques, 
mission statements, recommendations for curricula and/or teaching changes, assessment criteria, forms, and department 
calendars. All binders must contain departmental assessment plans. To assist departments in compiling appropriate data, a 
number of forms have been created, notably a use ofoutcomes assessment form and an annual report form. These forms prompt 
departments for specific data and ask them to analyze their assessment efforts. Ultimately, all academic departments are 
required to give on annual presentation to summarize their assessment activities and to present their binders to the 0 utcomes 
assessment committee. Instructors have found these sharing sessions helpful. 

In addition to Individual departmental plans, binders, and calendars, the outcomes assessment committee also maintains a 
college-wide OA assessment calendar {containing dates for advisory board meetings, department-wide assessments, and the 
like) and a master outcomes assessment activity bindertbat contains the plan for outcomes assessment of student learning at 
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Ranker) Technical College and the outcomes assessment model. The plan and model specify how assessment is tied to the 
mission of the college, set out specific goals of the program, and demonstrate how the circular process of assessment includes 
feedback loops to improve student learning. The ultimate goal is to transfer data from all of the binders into a recently created 
college-wide outcomes assessment databasethat is available on the institution’s intranet. This database will eventually contain 
all assessment data at the college, thereby cutting down on paperwork and making assessment information easily accessible 
to all faculty. 

While not conducted on a regular basis, ^oco/tydeve/opmenfsess/ons on assessment topics have been held numerous times. 
Sponsored bytheoutcomes assessment committee, these sessions initially assisted departments in developingtheiroutcomes 
assessment plans.Since then severalsessionshave been held, the latestbeingafacultypresentationbyOAleadership reviewing 
the assessment activities of the 2001-2002 academic year. 

Another strategy that has been successful in fostering communication about outcomes assessment has been the publication of 
a newsletter for faculty dealing with OA issues, both internally and externally. The OA Flash is the work of two OA committee 
members who keep abreast of assessment activities in thecollege and who research interesting events at other institutions. The 
newsletter is issued several times during the academic year. 

Staff are often conspicuously absent when dealing with outcomes assessment; however, some attempt has been made to inform 
them about the purpose and processes of assessment through the college’s employee orientation system. All new faculty and 
staff are required to meet with administrators, staff department heads, and some academic department heads to become 
familiar with howthe institution is run. A meeting with the chairperson of the outcomes assessment committee has been added 
to this procedure, and new employees also receive an Outcomes Assessment Question and Answer brochure to augment the 
information. 

□ Students 

The goal in involving students in assessment is not necessarily their active participation in the process, but rather to help them 
understand how assessment affects them and improves their learning. To that end Ranken has taken some modest steps to give 

students this information. Strategies include a written desc/'/pf/ono/'oofcomesossessmenf/nf/resfodenf/rondboo/rand regular 

articles about assessment activities in the RTC student newspaper. A more direct technique involves participating in semi- 
annual spirit days mtU an assessment information booth and an activity to arouse student interest, such as a raffle. Adisplay 
cose in a prominent location contains information about assessment and includes student projects and testing data. In the 
2002-2003 scademic year the faculty have also begun to include sfofemenfs about assessment in their syllabi, and instructors 
are encouraged to address the topic in their orientation sessions. While further work needs to be done in this area, the changes 
have been moderately successful in giving students an appreciation of how assessment can improve student learning. 

□ External Constituents 

Ranken Technical College has a long history of being connected to the industries that hire its students. Each department has an 
advisory board composed of recognized leaders in their field, and meetings are generally held two to six times a year. Advisory 
boards are an important means for technical departments to obtain feedback about the appropriateness of curriculum, new 
equipment, and industry trends. Additionally, the boards provide departments with up-to-date information on opportunities, 
product changes, and innovative techniques in their field. Several years ago outcomes assessment issues were added to the 
agendas of the advisory boards and, since board members are already committed to student academic achievement and often 
make curriculum suggestions, this has proved to be a useful technique. 

Along with departmental advisory boards in the majors, Ranken has recently formed a general education advisory board that 
deals with general education outcomes and other issues. The committee originally began in 2001 as a focus group examining 
general education goals and outcomes for the college’s first bachelor’s program, approved by the Higher Learning Commission 
in 2002. Since then it has continued to meet on a more formalized basis, now with the addition of three external members from 
other institutions and industry. While other general education issues are discussed by the group, improving student learning 
continues to be the primary focus. 

Another project undertaken by the outcomes assessment committee is the development of surveys for two groups of external 
constituents, alumni and employers. A subcommittee for each survey was charged with developing a brief questionnaire on 
student outcomes. The surveys were mailed for the first time in fall 2001 and met with limited success. As expected, some 
questions need revising to make them more specific, and the subcommittees are now preparing the surveys for a second round 
of mailing. The college plans to mail these on a rotating basis, and it is hoped that after a number of years the data will assist 
academic departments assess their outcomes. 

The most successful strategy involving external constituents is undoubtedly the Share and Compare Roundtable, held forthe first 
timeinMay2oo2.0utcomesassessmentcommitteemembersanddirectorsfromanumberofareacollegesand universities were 
invited to spend a day at Ranken discussing assessment issues. Six institutions participated in the event. The day featured a 
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loose agenda: a briefwelcome speech, a tour, lunch, and mostly a lot of discussion and sharingof assessment issues, problems, 
and solutions. The roundtable was so successful that everyone who attended wanted to make this an annual event, and plans 
are underway for the next session. Perhaps the greatest benefit of the roundtable was the realization that other institutions are 
struggling with similar issues and that, while some have made greater progress, others are still focused on gaining faculty 
support and working on implementing assessment at their institutions. 



Conclusion 

While a lot has been accomplished in the last four years, outcomes assessment is still, and will continue to be, a work in progress. 
More work needs to be don eon involving and informing students, developing a viable general studies outcomes assessment plan, and 
getting meaningful feedback from alumni and employers. Academic departments will need to cooperate and work with each other, and 
the college will continue to share with and learn from other institutions. In order to sustain itself, outcomes assessment will continue 
to grow, change, and hopefully improve in the years to come. Ranken Technical College feels that it can meet this challenge and 
successfully work at enhancing student academic achievement since the biggest obstacle has been overcome-that of integrating 
assessment into the life of the college. 



Barbara Edwards is Library Director at Ranken Technical College in St. Louis, Missouri. 
Vincent Holtmann is Faculty at Ranken Technical College in St. Louis, Missouri. 

Kelley Raftery is Faculty at Ranken Technical College in St. Louis, Missouri. 

Brad Sparks is Dean of Instruction at Ranken Technical College in St. Louis, Missouri. 
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Assessing an Institution’s 
Outcomes Assessment Efforts: 
The Application of the 
Higher Learning Commission 
Assessment Matrix by Three Institutions 



Introduction 

A successful outcomes assessment program requires a long-term view and commitment. Therefore, it is necessary to have a process 
in place that ensures progress toward those long-term goals. As coordinators for outcomes assessment efforts at the institutional 
level, we have struggled with how to assess the progress of outcomes assessment. This assessment of institutional progress is 
necessary for not only reporting to the Higher Learning Commission but also for ensuring that institutional assessment efforts have 
not stalled and determining how to further enhance those efforts. Specifically, the different kinds of issues we have encountered 
include the following. 

o How can we ascertain whether the institution is making progress in its assessment of student learning? 
o How can we effectively communicate the progress of our outcomes assessment efforts/effects with accrediting agencies? 
o How can we assess assessment at the institutional level to model a “culture of evidence”? 

o How can we analyze assessment reports received from individual units (college, academic departments, etc.) to better 

understand the contribution and progress of outcomes assessment at our institution? 

o How can we draft an institutional report about outcomes assessment that is a critical analysis rather than a summary of 
activities? 

in this session, three institutions will share how they have attempted to address these issues by using the Higher Learning 
Commission’s Assessment Matrix (formerly known as the Levels oflmplementationy as a guide. 



The Assessment Matrix \s an addendum to the Higher Learning Commission’s Handbook of Accreditation, It was designed to “assist 
institutions in understanding and strengtheningtheir programs for assessment.” The matrix consists of two dimensions: three levels 
of implementation by four patterns of characteristics. The three levels of implementation are “Beginning Implementation of 
Assessment Programs,” “Making Progress in Implementing Assessment Programs,” and “Maturing Stages of Continuous Improve- 
ment.” The four patterns are institutional culture, shared responsibility, institutional support, and efficacy of assessment. It is 
assumed thatthese patterns are dynamic and fluid across an institution and build upon one another. Institutional culture focuses on 
collective/shared values and mission. Institutional support involves the resources and administrative structures for the outcomes 
assessment process. Shared responsibility emphasizes the importance of assessment’s being a faculty-owned process that involves 
students and has strong administrative support. Efficacy of assessment encompasses the totality of all assessment efforts. Within 
each pattern for each level of implementation, benchmark descriptors of progress allow for documentation of progress over time. 
These descriptors emerged from analysis of content found in team reports and therefore are to serve as descriptive rather than 
definitive “markers of the progress institutions have made in developingtheir assessment programs.” 
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For each of our three institutions, these levels aided in framingvital assessment characteristics. However, each institution identified 
how the Assessment Matrix could be applied given our unique campus cultures and the maturity of assessment efforts at each 
university. As the followingthree descriptions illustrate, each university applies theAssessme/7f/Wafr/xin waysmostsuitableforits 
culture and structure. 



University of Nebraska-Lincoln 

AHheimeihe Assessment Matrix ms first released in 2000, the University of Nebraska-Lincoln (UNL) had just completed a Higher 
LearningCommission institutional focus visit on assessment. Although thefocus team members found that UNL had a strongstructure 
and strong support for outcomes assessment, they requested a progress report in two years on how the assessment process was 
contributing to the improvement of student learning. Consequently, the fourth pattern of characteristics, efficacy of assessment, held 
particular interest for us. 

Because UNL is a highly decentralized institution, most assessment efforts occur at the level of the academic unit. Thus, we needed 
to find a way to communicate our progressasa whole ratherthan through thegreatvarietyofdifferencesthatoccurred across academic 
units. The descriptions for efficacy of assessment assisted usinmovingtoward this bigger picture, but we determined that we needed 
to identify an additional levelofdetailthatwould help us measure the extentto which we wereachievingthe characteristics described. 
Therefore, we applied theframeworkfor efficacy ofassessment by identifying seven indicators that paralleled the descriptionsforthis 
pattern of characteristics but also represented what our institution valued about outcomes assessment. 

Once we had identified the seven indicators, UNL decided to capitalize on the wealth of information provided in annual assessment 
reports from academic units in a more formal and systematic way. Annual reports were a natural fit for use in collecting information 
about the seven indicators because as the reporting process evolved past mere development of outcomes assessment plans, we 
began to ask academic units to focus on two questions: 

1. How has the outcomes assessment process and evidence informed decision making? 

2. How has the outcomes assessment process led (or will lead) to the improvement of your educational program? 

Therefore, the number and content of the annual reports were analyzed to collect information relevant to the seven indicators so that 
progress on each could be tracked for the institution as a whole. 

For example, onecharacteristicfor efficacy of assessment is increased engagement amongacademicunitsand faculty in the collection 
and use of assessment results. To assess our progress on this characteristic, we would review an academic unit’s assessment report 
and make a determination about the extent of the impact of unit’s outcomes assessment activities on its academic program(s). This 
impact is analyzed at four levels, representing increasing levels of engagement with the assessment process: 

o Modification to assessment plans 

o Discussion of programmatic issues highlighted by assessment activities 
o Consideration of actions to improve student learning 
o Actions taken to improve student learning 

By annually determining the percentage of units engaged in the assessment process at each level, we could determine whether 
assessment efforts were increasingthe level of contribution overtime. It also allowed us to recognize the impact of initial assessment 
efforts, since much of what was learned in the beginning was about the assessment process itself. 

By tracking information on these indicators each year, we obtained a sense ofcontributions made to outcomes assessment effortsand 
whether those contributions were increasing each year. We started tracking this information in 1999 and have used it to enhance 
communication about our progress internally through UNL’s annual outcomes assessment reports and as external evidence for our 
Higher Learning Commission progress report due December 2002. Indirectly, the collection of information has assisted in clarifying 
expectations for assessment efforts and substantiating the contribution of outcomes assessment for those who question its value. 
The way we documented each of the seven indicators can be found in the appendices of the 2000 and 2001 annual assessment reports 
at <http://www.unl.edu/svcaa/priorities/assessment.html>. 



Oklahoma State University 

At Oklahoma State University (OSU), all academic programs are required by the state regents to have an outcomes assessment plan 
and to submit annual reports demonstrating their assessment efforts. The university’s assessment office and faculty assessment 
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council guide university-level assessments and also provide information and financial resources to academic programs to assist in 
developing effective program-level outcomes assessment. After a flurry of outcomes assessment activity following the initial 
implementation of the Oklahoma State Regents for Higher Education Assessment Policy i n 1993-1994, outcomes assessment in many 
programs slowed or stalled. A review of the outcomes assessment reports submitted by academic programs from 1995 through 1999 
showed that effective outcomes assessment was inconsistent, or inconsistently reported, among academic programs, and only a 
handful of programs demonstrated the type of progress articulated in the Higher Learning Commission’s Assessmer?fMofr/x. 

Dissemination of the Assessme/7t/Wofr/xencouraged the OSU assessment council to undertake peer review of outcomes assessment 
and work toward helping programs meet outcomes assessment expectations. IhQ Assessment Matrix provided a framework that the 
council used to develop its own scoring criteria appropriate for evaluating each academic unit’s assessment documents (plans and 
annual reports) as well as for providing constructive feedback about areas for improvement. The Assessment Matrixaiso states that 
“coaching and facilitating” were expected ofthe faculty assessment committee, which added strength and momentum to the review 
process. 

Every academic unit is reviewed every three years, and each review focuses on the program’s assessment plan and annual reports. 
The assessment office and the assessment council conduct the outcomes reviews jointly. The assessment office coordinates the 
logistics ofthe process and summarizes results for each program while small teams of council members conduct the reviews. The 
review criteria do not directly use the language of the Assessment Matrix but are based on principles described in that document (e.g., 
articulation of assessable student learning outcomes, use of direct and indirect assessment methods, use of assessment results for 
program development). The reviews emphasize the importance of helpful and constructive feedback, and follow-up information is 
provided to programs that request assistance. 

Reviews have been conducted since 2000, and the process has been successful in a number of ways. The reviews have provided a 
valuable opportunity for communicating basic assessment information and articulating what programs should be doing; faculty 
members relate that they appreciate the feedback. About 75 percent of programs have developed new assessment plans or 
demonstrated greater commitment to program outcomes assessment in their annual reports after the reviews began. The assessment 
council members themselves have become more aware of best practices in assessment, and the experience has enabled them to 
become more effective assessment leaders for our campus. Overall, the development of these reviews has been a critical step in 
moving the institution toward better assessment practices and improved assessment awareness. 

Information related to this process (copies ofthe evaluation forms and results summaries) will be provided at the presentation and 
can be viewed at <www.okstate.edu/assess/council>. 



Eastern Illinois University 

Like the University of Nebraska-Lincoln and Oklahoma State University, Eastern Illinois University (ElU) found the Assessment Matrix 
particularly useful at the institutional level for identifying the progress of assessment efforts and for developing a shared language 
when reportingto internal and external constituencies. As our assessment efforts moved from assessment of general education, an 
institution-wide effort, to the program and departmental levels, we found that we could adapt the matrix for use at this more specific 
level. The matrix aided in cutting across discipline-specific barriers in order to develop a reporting process and feedback loop suitable 
for all academic departments and comprehensible by all constituencies. 

Like many institutions. Eastern began its assessment efforts at the institutional level with a faculty committee focused on examining 
our mission and our general education to develop learning outcome goals for all our graduates. The impetus for this focus on 
assessment came from the 1995 “Report of a Visit to ElU” from the NCA team, who noted that “the campus has lost its early leadership 
edge and momentum in assessment activities. Specific steps must be taken to ensure full implementation of assessment plans with 
a clear focus on student learning outcomes.” 

Once an institution-wide assessment plan was established, it became clear that discussions similarto the ones that had taken place 
at the institution-level would be needed at the program level in order to encourage faculty to include assessment as part ofthe major 
and minor programs. Levelsof implementation were varied by college and discipline. Support and development opportunities were 
needed to provide advice and a framework forfaculty to organize, develop, and execute assessment plans based on student outcomes 
that could be embedded in established programs or used in the creation of new programs. 

In academic year 1999-2000, academic departments were asked to complete departmental assessment summaries. These summa- 
ries ask department chairs or assessment coordinators/teams to complete the following five sections; 

o Program (Major/ Minor/Concentration) Objectives 

o Assessment Measures (who, what, where, and how assessed) 
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o Program Expectations for Students* Learning Objectives 
o Results from the Current Academic Year 
o Description of the Feedback Loop and How Results Will Be Used 

All graduate and undergraduate programs at Eastern were required to submit a summary to the vice president for academic affairs. 
With a formal reporting structure in place, many departments were encouraged to create faculty-driven assessment committees or to 
turn to their departmental curriculum committees for discussions on major/minor objectives. 

To help departments with this effort, the committee for the assessment of student learning (CASL) shared the Assessment Matrix W\\h 
departments to encourage self-reflection, shared responsibility, and greater analysis at the department level about progress in 
assessingstudentlearningoutcomes. However, while CASLhad found the matrix usefulincreatinga university-wide assessment plan, 
the departments experienced difficulties in adapting institutional criteria to their work at the major/minor levels. To aid in their 
assessment efforts, a form using primary trait analysis was developed for departments. This tool provided criteria by level, 
corresponding to the categories reported for departmental assessment reports. In turn, those categories corresponded to the steps 
to creating assessment plans and developing a culture of assessment. The assessment office provides feedback on a response form 
to each department on its summaries with advice on ways to improve efforts; this feedback includes information on where each 
department falls in terms of the levels of Implementation. Copies of all these forms will be disseminated at the presentation and can 
be viewed at <http://www.eiu.edu/-'assess/departmentaLassessmenLplans.htm>. 

The response form encourages dialogue within departments as chairs, coordinators, and assessmentcommittees share the feedback 
with the rest of the faculty. The format also increases communication between those charged with providing support for assessment 
efforts and the faculty who develop assessment objectives and measures for their students. While departments are still at varying 
levels of maturity in terms of assessing student learning outcomes, the information gleaned from the summaries offers the institution 
a way to monitor progress forits internaland external constituencies while also providing useful information thatserves as an impetus 
to further discussion and shared responsibility and values. 



Summation 

The Higher Learning Commission AssessmenfAfflfr/x has enhanced the development of effective assessment at allthree institutions, 
yet interesting parallels and differences exist in our uses of the matrix. At each institution, the matrix was used to evaluate and 
enhance outcomes assessment reporting practices in academic units and make Inferences about the overall status of outcomes 
assessment at the university level. The University of Nebraska-Lincoln assessment office employed matrix concepts to review 
departments* outcomes assessment reports and used the review results to evaluate the university-wide status of outcomes 
assessment. At Oklahoma State University, the assessment office and faculty assessment committee conducted similar reviews; 
these reviews also provided an overview of outcomes assessment at the university level, but the focus of the OSU reviews was to 
provide feedback at the department level. Eastern Illinois University started by using the matrix information to establish and assess 
outcomes assessment processes atthe university leveland then used this approach to guide outcomes assessmentatthe department 
level. To varying degrees, each institution uses criteria developed from the concepts in the Assessment Matrix to assess efforts atthe 
department and university levels. Further, each Institution employs the matrix as a tool to communicate assessment expectations to 
faculty and facilitate quality outcomes assessment. 



Notes 

‘The Higher Learning Commission, “Chapter Reference A: Assessment of Student Academic Achievement: Levels of Implementation,** 
Addendum to the Handbook of Accreditation, Second Edition. March 2001, updated March 2002. 

"Ibid., p. 15. 

3 Ibid. 

Mbid., p. 25. 
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Assessing Student Learning: 
Elegance in Simplicity 

Gloria M. Rogers 



Introduction 

Faculty members are often thrust into the position of assessment coordinator or a similar position because they are exemplary 
teachers, have experience with statistics, education, or psychology, or just drew the short straw. However, they often do not have a 
mental model for howto create a systematic process for assessing student learning that is both efficient and effective. By developing 
a “lean and mean” systematic assessment process, the general workload on faculty can be reduced, the data will be more meaningful, 
and the resources required will be minimized. 



Use Models 

It is not necessary to start with a blank slate. Explore the models that have already been developed, and adapt those that are aligned 
with your institutional strengths as a beginning to the process. The AAHE 5 ^Pn/ 7 c/p/eso/'GoodPracf/ce^oMssess/; 7 g Student Learning 
(AAHE) is a must-read for institutions that are serious about understanding and implementing a robust assessment process. The 
critical elements of any model begin with the institution's mission statement and include the development of measurable learning 
outcomes, curriculum mapping, assessment strategies, evaluation, and feedback loops. The development of a model that is unique to 
the campus will help focus efforts in a systematic way, being sure that each process has an “owner” and is attended to in a timely fashion. 



Define the Terms 

It is important to create a “discourse community” of assessment. This can be done by defining the terms that are going to be used 
throughout the process. Unfortunately, a standard language is not used in assessment and evaluation. To avoid confusion and reduce 
the language barrier, it is critical that each institution decide what terms are going to be used, publicize them, and use them 
consistently when referring to the various processes. Table i lists some of the most common terms that are used. (Rogers, 2002). 

Table 1. Assessment Terminology 



Terms 


Definition 


Some Other Terms 
for Same Concept 


Objectives 


Statements that describe the expected accomplishments 
of graduates during the first few years after graduation 


Goals, outcomes 


Outcomes 


Statements that describe what students are expected 
to know and be able to do by the time of graduation 


Objectives, 
standards, goals 


Performance 

Indicators 


Specific, measurable statements identifying the 
performance(s) required to meet the outcome; 
confirmable through evidence 


Standards, criteria 
rubrics, specifications, 
metrics, outcomes 


Assessment 


Processes that identify, collect, use, and analyze data 
that can be used to evaluate achievement 


Evaluation 


Evaluation 


Process of reviewing the results of data collection and 
analysis to make a determination of the value of findings 
and action to betaken 


Assessment 
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Clarify the Outcomes 

One ofthe most common errors is to develop broadly stated learning outcomes statements without defining what the outcomes mean. 
The lack of definition makes meaningful data collection processes difficult, if not impossible. For example, a common learning 
outcomeforaninstitutionisintheareaofethics.Someinstitutionsevenstatethattheyhaveagoalthattheirstudentswill be “ethical.” 
What does that mean, and how will you measure it? Another institution may have as a learning outcome that students should 
“understand their ethical responsibilities.” What does this mean? What does “ethical responsibility” look like? How will students 
know when they have achieved “an understanding of their ethical responsibilities?” The learning outcome must be reduced to a 
manageable number of measurable performance indicators that will give focus to the assessment process and make it possible to 
measure and provide meaningful feedback to the institution on teaching strategies. For example, performance indicators for “ethical 
responsibilities” may be: 

1. Students should know a professional code of ethics. 

2. Students should be able to evaluate the ethical dimensions of a problem in their discipline 

These indicators reflect different levels of cognition and can be measured. More importantly, both students and faculty have a common 
understanding of what performance is required to demonstrate the outcome. The assessment methods also become more apparent. 
With the first indicator, students will need to demonstrate that they “know” a code of ethics. This could be the code of ethics for their 
discipline or a general code of ethics. Faculty would be able to incorporate a learning experience in the major or general education 
curriculum that would provide students with an opportunity to learn and demonstrate their knowledge. The data could be collected 
in the context ofthe course and used for institutional assessment purposes. The second indicator is at a much higher level of cognition 
and implies that they not only know the code of ethics but that they are able to evaluate the dimensions of an ethical issue within their 
discipline, compare and contrast possible actions, and come up with a recommendation for action. Articulation ofthe performance 
indicator enables faculty to focus the learning experiences that will provide students with an opportunity to practice and demonstrate 
what they have learned from the code of ethics in a meaningful context. 

Map the Curricula 

Once the learning outcomes have been defined, it is important to understand how the curricula support or are “mapped” to the 
outcomes. There are at least three dimensions to the mapping process. 

1. Is the outcome an explicit objective of a course? 

2. Do students get an opportunity to demonstrate their competence related to the outcome in a particular course? 

3. Do students get formal feedback on their performance on the outcome in a particular course? 

An example of a curriculum map appears in Table 2 (Rose-Hulman, 2002). Mapping the curriculum provides rich data that will enable 
the institution to strategically focus assessment efforts. It will also reveal where there are weaknesses in the curriculum. Assuming 
that the learning outcomes are ones that all students should be able to demonstrate by the time of graduation, each student should 
have opportunities throughout the curriculum (including their major) to learn, practice, demonstrate, and get feedback on their 
learning. What can be revealed through mapping is that in some majors students are never provided an opportunity to demonstrate 
their competence on the outcomes in the context of their major discipline. The results also might reveal that the outcomes are only 
found in elective courses so that a student could go through the course of study without taking a course in which they could learn or 
demonstrate competence on a given outcome. This could be the result ofthe “everybody’s job is nobody’s job” effect. In many cases, 
just the process of mapping the curricula reveals a great deal about the need to align curricula to what the institution says is important 
about student learning. 



Table 2. Curriculum Map. 



Institutional Outcome 


Objective 

Explicit 


Demonstrate 

Competence 


Formal 

Feedback 


Not 

Covered 


1. Recognition of ethical and professional responsibilities. 
View indicators or make a (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


2. An understanding of how contemporary issues shape and 
are shaped by mathematics, science, and engineering. 
View indicators or make a comme/?f (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


3. An ability to recognize the role of professionals in the 

global society. View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 



ERIC 



62 



BEST COlPT AVAMABLE 



Chapter 2. Assessment Processes/ 5^ 



Institutional Outcome 


Objective 

Explidt 


Demonstrate 

Competence 


Formal 

Feedback 


Not 

Covered 


4. An ability to understand diverse cultural and humanistic 
traditions. View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


5. An ability to work effectively in teams. 

View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


6. An ability to communicate effectively in oral, written, 
graphical, and visual forms. 

View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


7. An ability to apply the skills and knowledge necessary 
for mathematical, scientific, and engineering practices. 
View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


8. An ability to interpret graphical, numerical, and textual data. 
View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


9. An ability to design and conduct experiments. 
View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 


10. An ability to design a product or process to satisfy a client's 
needs subject to constraints. 

View indicators or make a comment (optional) 


□ Yes 


□ Yes 


□ Yes 


□ 



Develop a Timeline 

Faculty members are busy and have a very long list of important things that they want to accomplish. Assessing institutional learning 
for the purpose of institutional improvement is on the list, but not very close to the top. Caution must be taken not to develop an 
assessment system that will be so vast that it will collapse under its own weight. It is important to remember that data do not need 
to be collected on every student for every outcome every year. Develop a realistic time-frame where you focus on only a few (two to four) 
outcomes a year. The number will depend on how many outcomes you have and your “accountability cycle.” That is, what is the time 
span between reporting cycles to external agencies as well as internal accountability processes? For example, if you have nine 
outcomes, you may choose to focus on only three peryearand have a three-year cycle of assessment. Ifyou find that you consistently 
do well on one or more of the outcomes, you may decide to skip those and assess others more frequently while making improvements 
in the curricula. Remember, this process must work foryou, not for an external agency. You should be able to make modifications and 
multiple iterations in the process as needed. The important thing is that your decisions are evidence-based. 



Summary 

The key to efficient and effective assessment processes is to keep it simple but not simplistic. Don't try to measure too much too often, 
making the process too hard. You may have twenty-five things you think all students should know or be able to do by the time they 
graduate, but they cannot all be equally important. After you have made a decision about the subset that you are going to measure, 
you mayfind that asyou consistently do well on some,youmaywant to stop assessingthose and tackle others of importance. Everyone 
agrees that faculty members need to be involved in this process, but it is unrealistic to think that they will tolerate a system that 
requires them to alter their view of what it means to be a faculty member. Developing an effective and efficient system will invite them 
into the process and serve institutional needs as well. 
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Ensuring the Assessment 
Investment Pays Off: A Case Study 

Andrea Greene, Gail Mee, and Gayla Preisser 



The Assessment Problem: Some Background 

What do students actually learn across their several years of higher education?... Given all the assessment activity of the past 
decade...what are we now able to report from systematic assessment evidence about students' cumulative gains in learning 
across their years of study? The answer remains ...much too little.” (Schneider, 2002) 

Marchese (2000) poses several problems associated with the assessment movement, amongthem that assessment goes againstthe 
grain of academic culture; faculty perceive that outside entities want to “get the goods” on them; assessment practice is perceived 
as soft and lacking in scientific rigor; and the questions of assessment are not only uninteresting, but are dangerous. According to 
Marchese, these factors are not just problems, but “pathologies” plaguing the assessment movement. No wonder colleges and 
universities are having trouble developing and launching successful assessment initiatives; no wonder we still know much too little. 

At Mesa Community College (MCC), over the past eight years, the faculty and administration have made a significant investment in 
meeting the considerable challenges of the assessment problem by developing a credible program to assess student learning 
outcomes. As a result of these efforts, MCC is one of few colleges nationally that can answer the question, “What do students actually 
learn here?” Assessment results overtime show clear evidence of student learning in the college's general education and career and 
technical programs. These results also reveal areas forimprovementthatare used by faculty, departments, and the college as a whole. 
Using Mesa Community College as a case study, this paper addresses how colleges can invest their financial and human resources 
to ensure their student assessment programs are credible and sustainable and, most important, that results are used by the faculty 
to improve teaching and learning. 



MCCs Assessment Program: An Overview 

The Mesa Community College program to assess student learning outcomes is a college-wide initiative designed to measure and 
document the degree to which students attain specific learning outcomes valued and defined by faculty. The program is the result of 
a close collaboration among faculty, administration, and staff and has been recognized nationally for use of good assessment 
practices. Student assessment at MCC has become a cornerstone of the academic program and is an important part of evaluating the 
overall effectiveness of the college in achieving its mission. 

The college’s first assessment plan was written in 1994. Since that time, the program has evolved through numerous phases of 
development, implementation, and refinement. The phases include development of student learning outcomes; design, pilot testing, 
and revision of assessment tools, procedures, and data collection methods; and, most recently, a strong focus on the use of 
assessment results. Student learning outcomes are assessed for general education and in career and technical programs. 

Through the use of a cross-sectional research design, the performance of samples of students who are beginning programs of study 
in general education and career/technical programs is compared to the performance of students completing those programs. 
Assessment data are collected once annually during a spring assessment week. Through a randomization process, each student in 
the sample completes only one of several assessment measures. 

Faculty members use the results of student outcomes assessment to identify and document strengths in college programs, identify 
areas for program improvement, and ultimately improve student learning. Dissemination of assessment results and integration of 
results into departmental and college planning have become increasingly important as the program matures. 

The assessment structure at Mesa is highly collaborative, with faculty, administration, and staff working together to continuously 
improve, sustain, and institutionalize the assessment program. The faculty senate student outcomes committee (SOC) is a standing 
facultycommittee charged with making decisions and recommendations related to all aspects ofstudent outcomes assessment at the 
college. The committee is comprised of more than twenty full-time faculty and four ex-officio members (the dean of instruction and staff 
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from the office of research and planning). Full-time residential faculty members receive reassigned time in the fall and spring 
semesters to serve as chair and chair-elect of the SOC and to lead the assessment initiative. 



What We’re Learning About Student Learning: The Results 

The Mesa Community College program to assess student learning has helped to answerthe question, “Are students learning as a result 
of their experience at the college?” Evidence of student learning is clear in both the general education and the career and technical 
assessments; patterns of evidence across measures and over time reinforce the viability of the results. 

The program has engaged faculty, students, and administrators in discussions about the desired outcomes from a general education 
program of study and from participation in a variety of career and technical programs of study. MCC faculty recognize assessment as 
one important tool to document the successes of college programs in terms of student learningand also to improve student learning. 
Through the efforts of the student outcomes committee as well as individual departments across the college, strategies for using 
assessment results are being formally proposed. These include faculty development, faculty collaboration across disciplines, and 
involving new and adjunct faculty members through a variety of means. Student assessment and its results strongly reinforce the 
college's commitment to its primary strategic direction-a focus on learning. 



Ten Characteristics of Effective Assessment Programs 

There is no standard model for developing a good assessment program. Just as each college and university lays claim to its own 
uniqueness, so will each develop its own unique assessment program. But there are organizational characteristics that can make the 
success of these programs more likely. These conditions form the foundation for effective assessment programs, and they apply to 
colleges and universities large and small, urban and rural, public and private. The characteristics were developed as a result of what 
has been learned over almost a decade of assessment practice at Mesa Community College, and the efficacy of the characteristics has 
been reinforced through recent empirical research, especially that of Peterson et al. (2002) and others. 

1. Assessment is driven by college values. Assessment of student learning begins with the stated mission and values of the 
institution. At MCC, for example, the mission addresses transfer education, career preparation, continuous learning, 
developmental education, and economic development. Learning is one of the four core college values. These elements, then, 
form the foundation for and drive the college's approach to student outcomes assessment. 

2. The college makes a long-term commitment to assessment. Good student outcomes assessment is a process, not an event. 
It is ongoing, not episodic. In successful assessment programs student learning is measured overtime, and the language and 
practice of assessment become part of the college culture. Assessment requires a long-term commitment from the entire 
college community, from the president through the administration, the faculty, and the staff. 

3. The administration understands assessment and believes in its value. Superficial administrative support for assessment 
is not sufficient for sustaining a successful assessment agenda. If college leaders simply pay lip service to student outcomes 
assessment i n order to satisfy accreditation organizations, the assessment program will not be viewed by others as vital and 
important. Administrators must themselves be fluent in assessment practice and able to articulate its worth and demonstrate 
that they believe in its value. 

4. Faculty lead the program and own the results, just as college programs and curricula are clearly the purview of the faculty, 
so too is student outcomes assessment. An assessment program that is developed by the administration, staff, or any entity 
other than the faculty will fail. Faculty members need to be responsible for defining learning outcomes, determining the best 
methods for assessing those outcomes, providing input into data collection efforts, and using the assessment results to 
improve teaching and learning. 

5. Technical expertise and support are provided. The technical aspects of student outcomes assessment can be daunting. 
While faculty members are experts in their subject matter, most are not experts in educational research methods and 
practices. In addition, expecting faculty to be responsible for the management and coordination of the people, tasks, and 
materials required to sustain an assessment program is unrealistic. With the technical support of an institutional research 
office, faculty members can focus on assessment by contributing their expertise with teaching, curriculum, and student 
learning. 

6. Learningoutcomesaredefined programmatically. Educatorsaregood at assessinglearningoutcomesatthe classroom and 
course levels, but often are not familiar with the concept of defining and assessing student learning outcomes at the program 
level. A “program” implies any structured educational activity with specific goals and outcomes. A program might represent 
a course of study, for example, a general education program. It might be defined as a degree, for example, a bachelor of arts 
degree. Sometimes, programs are synonymous with departments, as in the case of most nursing programs. Once a program 
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is defined, specific learning outcomes expected to result from completion of the entire program can be defined by the faculty 
members who are collectively responsible for it. 

7. Measurement tools align with learning outcomes. Assessment measures should align directly with the learning outcomes 
defined by the faculty. Sometimes, externally developed assessment tools can be identified that are well aligned with the 
outcomes. Often, however, faculty will not be satisfied with published assessment tools and prefer to develop their own. In 
this case, it is important to systematically pilot test the faculty-developed measures, conduct data analyses to determine the 
validity and reliability of the measures, and make appropriate revisions based on the analyses. 

8. A viable research design and sound research methodology are used. When launching a student assessment initiative, it 
may be tempting to start assessing everything and everyone in the college. This approach will yield a large amount of data, 
but will tell little about what students are learning as a result of a defined program of study. By selecting a research design 
that is appropriate for the particular context of the college or university, the program will be much more likely to result in 
assessment data that are both valuable and credible. Exactly what will be assessed? Who will be assessed and when? How 
will the measures beadministered?The useofcommonly accepted research methods will provide thecollegecommunitywith 
assessment data that are valuable for making judgments about whether students are attainingthe learning outcomes valued 
and defined by the faculty. 

9. Results are used by faculty to improve learning. A good research design and data collection plan will provide good data for 
analysis. Once analyzed, results should be disseminated at levels appropriate to the various constituents of the college 
community. Faculty members, both individually and within their departments, should have ample opportunities to examine 
the assessment results and engage in meaningful discussions about them. They should be encouraged to make recommen- 
dations for changes in the curriculum or in teaching approaches that might improve student learning. Students and 
community members are also very interested in assessment results because they provide evidence about the effectiveness 
of an institution’s programs of study. 

10. Assessment is linked to college planning. Departmental and college-wide planning processes should be in place to 
accommodate the use of results from student assessment programs. At MCC, for example, departments review assessment 
results each fall when preparing their departmental initiatives for the following academic year. They propose departmental 
initiativesthataddressareasforimprovementinstudentlearningoutcomes,andidentifytheresourcesneededtoaccomplish 
those initiatives. In addition, a cross-disciplinary “results outreach committee” has been formed to make recommendations 
for college-wide initiatives, such as faculty development, based upon assessment results. 

The Assessment Investment 

Developing, implementing, and sustaining a credible student outcomes assessment program requires a considerable investment by 
the faculty, staff, and administration of the college or university. If the program is successful, that investment of human and financial 
resources will be viewed as well spent. At Mesa Community College, by applying the characteristics outlined in this paper, the 
assessment investment has paid off. Assessment has become regular part of the way the college does business and is one of the 
cornerstones of the academic program. The characteristics provide a foundation upon which other colleges and universities can build 
successful assessment programs. 
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Evaluating Assessment: 
Turbulent Flight in the Life of an 
Assessment Evaluation Pilot Process 

Janice Collins and josh Bullock 



Introduction 

Moraine Park Technical College in Fond du Lac, Wisconsin, offers a variety of certificates, associate degrees, and technical diplomas 
in occupational and vocational areas. As a learning college we redefined curricula and implemented an assessment portfolio process 
to meet the needs of employers and to monitor and improve student learning. This paper will take you through the turbulent flight of 
a pilot evaluation process of the assessment plan at Moraine Park. 



Pre-flight Planning 

In 1999, Moraine Park Technical College began pre-flight planning for assessment of student learning. Moraine Park Technical 
College’s traditional use of a performance-based curriculum directed the selection of learning portfolios as the method of assessing 
student learning. The challenge with any assessment process is to quickly and efficiently integrate this transition into the college’s 
operatlonsandcoreculture.lnitially,theHigherLearningCommission’s“Levelsoflmplementations— PatternsofCharacteristics”was 
used as the guide for Integrating assessment throughout the college. As a pilot Is required to file and follow a flight plan for reaching 
the final destination, students are similarly required to plan the collection of artifacts to be used as evidence of learning in compiling 
an assessment portfolio. This flight plan Is a requirement for all students’ final destination— graduation. 



The Turbulent Takeoff 



lnAugust2000 Moraine ParkTechnIcalCollege’sassessmentplantookflight after severalyears in development. Students who Started 
coursework toward a technical diploma or an associate degree at that time were required to collect artifacts to Include in their 
portfolios. In June 2001 the first portfolio evaluations were conducted on the first group of graduates. In the review of these first 
submitted portfolios, It became evidentthat many students’ flights experienced extreme turbulence in the form of miscommunication, 
misunderstanding, and lack of direction from faculty and staff. This meant that many students were off course In reaching their final 
destinations, and it left faculty confused and frustrated. 



Also, In June 2001 a full-time outcomes assessment associate was hired to captain the process and seek improvements. Over the 
course of the next year measures were taken to Increase communication and clarify the process for faculty and students. Including 
providing students with portfolio checklists for compiling artifacts and guidelines to assist them in organizing their portfolios. In 
addition, an effort was undertaken to create college-wide awareness of assessment through faculty forums, assessment newsletters, 
and “Sparkles” recognition awards to anyone in the College community for outstanding contributions to the assessment process. 

lnJune2002,thesecondgroupofstudent portfolios was reviewed. These portfolios were noticeablyimproved from the 2001 portfolios, 
in that students wereableto link reflections on learningto artifacts as evidence of learning. Awareness of assessmentthroughout the 
college also appeared to have increased. Much of the information gathered was anecdotal with qualitative value. Quantitative data 
to measure effectiveness of the assessment process were necessary to validate that the assessment plan was on course. 



By 2002, two years into flight, the assessment plan directed that it was time to officially gather quantitative data to track course and 
direction of assessment within the college and to gather baseline data to measure “where we stand today.” A pilot evaluation survey 
was launched to follow the assessment process. Much like a hot air balloon following the air streams, we were frequently blown off 
course and often accused of being full of hot air, but we needed proof to ensure that we were still on course. With the “Assessment 
Matrix-Patterns of Characteristics” (The Higher Learning Commission, March 2002) as the map, our direction and goal were clear. 
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The goal was to determine which level of implementation Moraine Park’s assessment plan was at in fall 2002: Level One, Beginning 
Implementation of Assessment Programs; Level Two, Making Progress ImplementingAssessment Programs; or Level Three, Maturing 
Stages of Continuous Improvement. Using the size of the institution and the scope of the research as our anemometer, we realized 
that a written survey instrument would be the most manageable way to conduct the research. 



Reaching Cruising Altitude 

Once take*off was complete and cruising altitude was reached, seatbelts were unclasped, and the team began to design the survey 
instrument. Questions were developed using the characteristics and descriptors provided in the “Assessment of Student Achieve* 
ment: Levels of Implementation” published by The Higher Learning Commission (March 2002). The pilot, Janice Collins, and the 
navigator, josh Bullock, solicited feedback from many ground and flight crews (faculty and staff) throughoutthe process to ensurethat 
the metrics addressed Moraine Park's learning college principles. Questions were framed to allow respondents to state their levels 
of agreement with each assessment related statement. A four*polnt Likert scale was used, with choices of strongly agree, agree, 
disagree, and strongly disagree. In addition, a response of “not sure” was included to prevent forcing respondents from stating 
agreement with unfamiliar statements. Because items were included from each ofthe four sections of “Patterns of Characteristics,” 
the final product was a five*page, scanner*based survey instrument consisting of seventy*eight items. 

Faculty who act as assessment advisers recommended surveying all groups affected by assessment in the college. Thus, a sample of 
545 people was drawn, consisting of 181 support staff, 79 managers/confidential employees, 145 faculty members, and 140 call staff 
(adjuncts). 

The survey was distributed via intercollege mall in early December 2002. A modest (edible) Incentive was used to encourage 
respondents to complete and return the survey to their respective campus libraries within a two-week window. Due to the short time 
frame for fielding the survey, reminder letters were not sent out. 



Cleared for Landing 

As our journey neared the end of its first leg, we radioed the control tower and prepared for landing. Was our connecting flight on 
schedule, or would we meet with unexpected delays? Atotal of 210 responses were returned by the survey close date, for a respectable 
response rate of nearly 39 percent. The baggage carts quickly arrived and whisked the surveys off to the College Research Services 
office for scanning into Teleform’ and data analysis using SPSS. The landing had been smooth; it was time to deplane and debrief. 



Post-flight Debriefing 

Given the turbulent takeoff and lengthy flight, the pilots were anxious for a debriefing. Is the college assessment plan still on course? 
Or have the prevailing winds pushed us farther from our destination? Data were analyzed in aggregate as well as being broken out by 
employee groups, campus location, length of employment at the college, and whether respondents had participated in a professional 
development session on Moraine Park’s assessment plan. Some critical debriefing points are presented in the tables below. Note that 
the debriefing points address only the perceptions ofthe faculty group and do not include the perceptions ofthe staff, managers/ 
confidential employees, and call staff (adjuncts). It should also be pointed out that in many instances the faculty’s level of agreement 
with the assessment statements was lower than expected, not as a direct result of high levels of disagreement but because many 
respondents simply stated that they were “not sure.” 

In Table i it can be seen that nearly two*thirds of faculty view assessment as part of the college’s culture and valued across the 
institution, but they do not recognize assessment as a resource to improve teaching and learning, and believe that students view 
assessment in the same way. Ironically, faculty members believe that managers view assessment findings as a source of improving 
teaching and learning. 
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Table i: Faculty Perception of Institutional Culture 



INSTITUTIONAL CULTURE 


/^ree 


Disagree 


Faculty perceptions only* 


strongly Agree 


Strongly Disagree 


Collective Shared Values 


Faculty view assessment activities as a part of MPTC’s culture or 
“way of doing business." 


59.4% 


29.7% 


Faculty view assessment activities as a resource and tool to be 
used in improving student learning in all programs. 


29.2% 


55.4% 


Students view assessment activities as a resource and tool to be 
used in improving student learning in all programs. 


16.9% 


64.6% 


Assessment of student learning is valued across the institution 
and the programs it offers. 


49.2% 


41.5% 


Assessment of student learning is an integral component of 
occupational programs. 


71.9% 


14.1% 


Managers regard assessment findings as a source of 
knowledge essential for continuous improvement in instruction 
and program offerings. 


50.8% 


20.0% 


Faculty regard assessment findings as a source of knowledge 
essential for continuous improvement in instruction and 
program offerings. 


40.0% 


49 - 3 % 


Mission 


Assessment materials, documents, and reports reflect the use of 
assessment results to improve student learning. 


38.5% 


40.0% 



* Totals do not add up to 100% because they do not include the “not sure” response. 



Facultymembers perceive that a vast majority oftheir peers engage in assessment practices that provide evidence of student learning, 
yet do not speak publicly in support of assessment (Table 2). They believe that quantitative and qualitative assessment measures are 
aligned with course and program outcomes and are taking responsibility for ensuring that direct and indirect measures of learning 
are aligned with these outcomes. 

Facultymembers perceives that the assessment process receives outward support by top management. In addition, they believe that 
students are becoming more knowledgeable about assessment and are provided formal occasions to reflect upon theiracademicwork 
and their thoughts about their achievement of learning outcomes. Although student knowledge is increasing, more than 50 percent 
of faculty surveyed believe that students still know tittle or nothing about how the assessment process will be carried out, their role 
in its success, or how it could be useful to them. 
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Table 2: Faculty Perceptions of Shared Responsibility 



SHARED RESPONSIBILITY 


Agree 


Disagree 


Faculty perceptions only* 


strongly Agree 


Strongly Disagree 


Faculty: 


A majority of instructors engage in assessment practices 
that provide evidence of student learning. 


70.4% 


94% 


A majority of instructors speak publicly in support of assessment. 


12.7% 


57.1% 


Faculty members are taking responsibility for ensuring that 
direct and indirect measures of student learning are aligned 
with the learner outcomes. 


65.7% 


10.9% 


Quantitative assessment measures are aligned with course and 
program outcomes. 


67.2% 


12.5% 


Qualitative assessment measures are aligned with course and 
program outcomes. 


70.3% 


94% 


Administration and Board: 


The Vice-President of Teaching and Learning, Deans, Partners, and 
other administrators demonstrate their commitment to assessment 
through outward support of assessment personnel and activities. 


76.1% 


6.4% 


Students: 


Students are becoming knowledgeable about MPTC’s 
assessment processes. 


71.9% 


21.9% 


Student leaders educate their peers about the assessment process 
through conversations, public presentations, and written 
communications. 


21.9% 


21.9% 


Students know little or nothing about the assessment process: 
they do not understand how it will be carried out, their role in its 
success, or how it could be useful to them. 


51.6% 


39.1% 


Throughout their academic programs, students are provided formal 
occasions to reflect upon their academic work and express their 
thoughts about their achievement of learning outcomes. 


57.1% 


12.7% 


Students are regularly required to present explanations of how 
work assignments demonstrate attainment of course and 
program outcomes. 


40.7% 


23.3% 



* Totals do not add up to 100% because they do not include the “not sure” response. 



It is evidenced in Table 3 that faculty members are aware of the formal assessment processes and documentation in addition to 
perceptions of assessment’s integration into the curriculum. Also, faculty members agree that there is a good level of communication 
about assessment to the campus community and a belief in the commitment of the major process team to improvement, just over one- 
third of faculty respondents think that sufficient resources are budgeted to operate and sustain the comprehensive assessment 
process. About the same number of faculty respondents believe that the institution regularly collects data and evaluates the 
assessment process. 
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Table 3: Faculty Perceptions of Institutional Support 



INSTITUTIONAL SUPPORT 
Faculty perceptions only* 


Agree 

strongly Agree 


Disagree I 

strongly Disagree 1 


Resources: 


Sufficient resources are allocated in the budget to operate and 
sustain the comprehensive assessment processes. 


42.2% 


18.7% 


Structures: 


There is written documentation of the implementation of the 
assessment plan. 


777% 


3.2% 


Syllabi for courses and programs state measurable objectives for 
student learning and provide for the assessment of students’ 
academic achievement. 


76.6% 


7.9% 


The institution maintains a system of data collection that helps 
sustain an effective assessment process. 


37 - 5 % 


20.3% 


The comprehensive assessment process is evaluated regularly. 


37 - 5 % 


14.1% 


Members of the Outcome Assessment Major Process Team serve as 
advisers to individuals and teams working to develop or improve 
their assessment process and activities. 


68.2% 


14.3% 


Information about assessment activities and their results is 
communicated regularly to the campus community. 


54.7% 


29.7% 



* Totals do not add up to 100% because they do not include the “not sure” response. 



Faculty respondents overwhelmingly agree that student learning is central to the culture of the institution and that finding ways to 
improve it is ongoing (Table 4). But, faculty respondents do not believe that assessment results are currently being used for this 
purpose. This belief may be exacerbated by the fact that nearly 83 percent of faculty respondents agree that confusion exists regarding 
thedifferentpurposesand relationships among: faculty evaluation, assessmentofstudent learning, program review, and institutional 
effectiveness. Only 39 percent of faculty perceive the college to be publicly celebrating demonstrated student learning, performance, 
and achievement. 



Table 4: Faculty Perceptions of Efficacy of Assessment 



EFFICACY OF ASSIGNMENT 


Agree 


Disagree 


Faculty perceptions only* 


Strongly Agree 


Strongly Disagree 


Implementation of the assessment process is progressing at an 
expected rate. 


42.2% 


14.1% 


Academic programs are using assessment results. 


42.9% 


14.3% 


Confusion exists regarding the different purposes and relationships 
among: faculty evaluation, assessment of student learning, 
program review, and institutional effectiveness. 


82.8% 


4.7% 


Academic programs are collecting. Interpreting, and using the 
results obtained from assessing student learning. 


35.9% 


21.9% 


Student learning is central to the culture of the institution. 


75.8% 


17.8% 


Finding ways to improve student learning is ongoing. 


74.6% 


12.7% 


Programmatic benchmarks are established against which students’ 
learning outcomes are assessed. 


42.9% 


9.5% 


The institution publicly celebrates demonstrated student learning, 
performance, and achievement. 


38.7% 


20.9% 



Totals do not add up to 100% because they do not include the “not sure” response. 



66 /A Collection of Papers on Self-Study ond Institutionol Improvement, 2003 - Volumes: Promoting Student Leoming ond Effective Teoching 



Preparing to Meet Our Connecting Flight 

Debriefing notes in hand, the pilots begin to make minor adjustments to flight plans in anticipation of the second leg of the journey, 
relieved that they are still on course and on schedule. Until the data are further analyzed, no major changes in the assessment plan 
will take place, only immediate tweaks to react to the changing atmospheric conditions. Several short-term actions have been 
Identified, including the following. 

o Devise action plans to Increase faculty's use of assessment as a tool for improving student learning in all programs. 

o Encourage faculty to publicly speak out in support of assessment by sharing their successful assessment practices with the 
learning community. 

o Increase efforts to educate students on the purpose and value of assessment in improving their own learning experience. 

o utilize educational tools to eliminate faculty's confusion amongfaculty evaluations, assessment of student learning, program 

review, and institutional effectiveness. 

o Maintain efforts to Incorporate the assessment process into the college's “way of doing business.” 

o Continue faculty involvement as members of the Outcome Assessment Major Process Team for decision-making processes 
and communicating assessment principals throughout the college community. 

o Maintain management's outward support of assessment processes. 

o Continue our practices that align quantitative and qualitative assessment measures with course and program outcomes. 

This document provides an overview of our flight plan. There is still much analysis to present in regards to management/confidential 
employee, staff, and call staff perceptions concerning assessment. In addition, the effects of assessment training, campus location, 
and length of service with the college will be addressed in the presentation session and subsequent roundtable discussion. 
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Gauging the Level of 
Faculty Participation in 
Student Academic Assessment 

Steve Boettcher, Bryan Tippett, and Clay Goodman 



Introduction 

The levels of implementation model has been used by many institutions of higher education to self-assess the progress of their student 
academic assessment programs (Lopez, 1999). The pattern of characteristics provided for each level of implementation helps 
determine whether the assessment process shows beginning implementation (Level One), some implementation (Level Two), or 
ongoing implementation (Level Three). The language of assessment requires a common point of reference, and consultant-evaluators 
may use an institution's levels of implementation analysis to facilitate discussion about the assessment program. 

During a recent accreditation team visit, Estrella Mountain Community College found that neglecting to gauge the level of faculty 
participation may undermine an otherwise successful assessment program. The question “What percent of the faculty have bought 
into assessment at your college?” may appear to be fundamental, but the levels of implementation matrix does not provide a clear 
answer. Although Estrella Mountain Community College conducted levels of implementation analysis, no college information or 
evidence provided appropriate documentation forthe level of faculty participation in assessment. This paper discusses the problems 
that result when faculty buy-in is not gauged, and presents ideas for achievingthe highest level of assessment program implementation. 



Background 

Estrella Mountain CommunityCollege is the newest oftheten Maricopa CommunityCollegeslocated in metropolitan Phoenix, Arizona. 
Thecollege enrolled 7,300 students lastyearand expects a lo percent increasethisyear. At present, there are fifty-one full-time faculty 
members. 

In 1996 Estrella Mountain applied for initial accreditation with the Higher Learning Commission. As a result of the initial evaluation 
team visit, the college was required to submit a progress report on student academic assessment. The Higher Learning Commission 
approved the 1998 progress report, which updated the college's plan for student academic assessment. 

The student academic assessment plan was updated again in 2001 as part of Estrella Mountain's preparation for its first ten-year 
accreditation team visit in February 2002. In addition, the college used the levels of implementation matrix (Lopez, 2000) to conduct 
a self-assessment of the student academic achievement process. Compared to the patterns of characteristics provided for each level, 
the self-assessment indicated that the assessment program at Estrella Mountain was at Level Two, with some characteristics at Level 
Three. 

In February 2002 the visiting team recommended continued accreditation with the next comprehensive evaluation in ten years and 
no interim monitoring. However, the visitation team pointed to the lack of full faculty participation in student academic assessment 
as an issue of concern. The visitation team report commented that “approximately one-half of faculty who teach general and/or 
transfer-education courses have not truly ‘bought in' to the process [ofstudent academic assessment].” The college understands that 
the remark was made by a faculty member of the Student Academic Assessment Committee either as a casual conversation with a 
visitingteam member or during one of the visiting team meetings with college faculty. Regardless, no college information or evidence 
existed to either substantiate or refute this claim. 

The claim about the level of faculty support had an unfortunate result. It allowed the readers' panel to disagree with the findings of 
thevisitation team and recommend a focusvisit on assessment. Confusion was warranted because the college did not gauge the level 
offaculty participation with appropriate documentation. When Estrella Mountain appealed, the reviewcommittee upheld the visiting 
team's recommendation based on additional information not available to the readers. While the appeal was successful, the college 
found that neglectingtogaugethe level of faculty participation could have undermined an otherwise successful assessment program. 
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If an institution is at Level Two with some characteristics at Level Three, do you know what percent of the faculty have bought into 
assessment? The term faculty buy-in appears in the 2002 levels of implementation matrix (Lopez, 2002), and faculty involvement is 
an essential component of any assessment process. However, if full faculty participation in the assessment process may be an area 
of concern, then Estrella Mountain’s self-study experience suggests that an appropriate means of documenting the level of faculty 
participation in student academic assessment is needed. 



Problems with Documenting Faculty Support for Assessment 

An approach that gauges individual faculty members’ support for the student academic assessment process faces major hurdles. 
Faculty resistance to an employee evaluation that includes the assessment process is strong. At Estrella Mountain, the assessment 
effort has deliberately been kept separate from the employee evaluation process. The residential faculty policies do not require 
participation in the assessment program, and an attempt by the college to create this requirement would be perceived as a violation 
of current policy. 

When preparing documentation for the appeal, there was hope that a questionnaire that gauged faculty support for student academic 
assessment might provide useful information. An anonymous survey was administered by the faculty senate with a 90 percent 
response rate (forty-six out of fifty-one). The first question was 

Classroom level assessment involves measuring what students learn and using the results to improve the 
instruction in your particular class. How do you respond to the following statement? “My activities support 
classroom level assessment at Estrella Mountain.” 

One hundred percent of the faculty responded that they “strongly agree” or “somewhat agree” that their activities support classroom 
level assessment. The second question was 

Institutional level assessment involves measuring what students learn and using the results to improve the 
instruction college-wide. Two student abilities that Estrella Mountain assesses at the institutional level are 
critical thinking and communication. How do you respond to the following statement? “My activities support 
institutional level assessment at Estrella Mountain.” 

Ninety-six percent of the faculty responded that they “strongly agree” or “somewhat agree” that their activities support institutional 
level assessment, and 4 percent reported that it was “not applicable.” The high percentages that indicated support for the assessment 
process as well as the self-reporting aspect of the survey made the results appear to be biased and/or false. It was concluded that the 
questionnaire was not an appropriate means of documentation, and the survey results were not included in the appeal to the review 
committee. 



Ideas for Gauging Faculty Support for Assessment 

After discussions with the faculty and the Student Academic Assessment Committee, Estrella Mountain has opted to pursue an 
approach that has received the unanimous support of the division chairs; namely, assessment will be incorporated in each division 
strategic plan. The plans will provide a description of each assessment activity supported by the division faculty, a rationale for the 
levelofimplementation thateachactivitysupports, a description of how the results will be measured, and a summary oftheoutcomes 
produced (if available). Results showing the percent of faculty involved with activities that support Level One, Level Two, and Level 
Three as per the division strategic plans would be included. In addition, individual faculty members will be encouraged to develop an 
assessment portfolio that documents support for assessment. These portfolios will be reviewed by the coordinator of student 
academic assessment and by two peers chosen by the faculty member. The purpose of peer review is to provide feedback and to 
improve the level of assessment program implementation by deepening the culture of collaboration. 



Conclusion 

By neglecting to gauge the level of faculty participation in student academic assessment, Estrella Mountain found that the levels of 
implementation analysis did not provide a satisfactory answer to the question "What percent of the faculty have bought into 
assessment at your college?” An appropriate means of documenting the level of faculty participation in the assessment process is 
needed. One approach may bethe use of division strategic plans to documentassessment activities, augmented by individual faculty 
assessment portfolios that support these activities. The percent of faculty involved with activities that support Level One, Level Two, 
and LevelThree as per the division strategic plans would be documented. Best practices for gauging the level of faculty participation 
in assessment have yet to be determined. 
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Are We There Yet? 

One Institution’s Assessment Adventure 

Anneliese Homan and Beverly Wilkerson 



Introduction 

State Fair Community College (SFCC) offers associate of arts and associate of applied science transfer degrees as well as career and 
technical certificates to approximately 3,500 students in a fourteen-county service area in rural Missouri between Kansas City and St. 
Louis. When the institution first received NCA accreditation in 1974, it was struggling to distance itself from the nickname “Plywood 
U,” which came from the cheaply constructed buildings housing its first classrooms and administrative offices. These buildings no 
longer exist. In fact, in only a decade the campus has gained four new multi-million dollar facilities funded by community donors* 
support, earned by faculty and staff who have established a reputation for academic integrity and quality service. 

Prior to NCA*s visit in 1999, most faculty agreed that assessment of student performance occurred each semester. If enough students 
failed to master an objective, more than likely something was wrong with the teaching method. Hence, the old one was altered or a 
new one was found. Many argued that good instructors conducted this process intuitively. For that reason, the suggestion that a formal 
plan should be created to document this process was met with polite disapproval from some and thunderous resistance by others. 
The reviewing team recommended that SFCC receive continued accreditation in 1999, but because most of the institution’s programs 
lacked plans to assess learning or produce essential data for making decisions, NCA requested a progress report in 2002. By the time 
the requested report was prepared last spring, however, all programs had developed plans and had completed or were in their first 
round of collecting data. Although the SFCC assessment committee still has some issues to address, it has made significant progress 
in steering its faculty onto the chosen thoroughfare and in the same direction, despite some mid-route detours and bouts of road rage. 

This paper chronicles the adventure. The session presenters, both student outcomes assessment committee members and 
department chairs, discuss the specific challenges the college has faced in its effort to emphasize the need for documenting 
assessment, not only as the practice has been traditionally conducted, butthroughthedevelopment of formal plans that communicate 
expected outcomes and identify a means for acquiring data to measure these outcomes. In turn, the presenters highlight the specific 
strategies used to help faculty members feel less like unwilling passengers and more like drivers with a clear sense of direction. 



How We Got Lost 



Perhaps there would have been less resistance toward the issue of formal assessment if a simple and precise plan for coordinating 
and implementing the assessment process had been presented to the faculty in the beginning. Prior to the site visit, only a few 
programs had addressed the issue of assessing student outcomes for program improvement. Of the twenty-seven programs, only the 
nursing, medical office assisting, custom applicator, and automotive and industrial maintenance programs had documented 
measurable competencies, albeit at various stages of specificity. By 1996, the general education program had established five goals 
that were being assessed throughout the curriculum; however, many faculty members had difficulty with this process. They couldn’t 
make the connection between the general education goals, program goals, and using the data to improve student learning. Faculty 
were also skeptical that their efforts to produce data would bemeaningfulorcontributeto and enhance student learning, and program 
coordinators felt that assessment was an additional responsibility not included in their job description. 



In the meantime, members of the assessment committee themselves had no clear sense of the direction the institution’s assessment 
process should take. When the self-study committee for assessment co-chairs looked to the administration for guidance, they were 
simply told to “study the process.” Because no one clearly understood the process, the institution was not able to clearly articulate 
a plan. Consequently, as the NCA review team noted in the 1999 report, assessment efforts were “unfocused.” In its comments, the 
team reported that assessment work was limited to the course rather than program level and appeared to be “a highly individualized 
activity rather than a group process to improve the institution” (Griego, et al., 1999, p. 17). The existing data often could not be tracked, 
orthey were inappropriate. For instance, the art department relied on an outside jurorto evaluate student workand provide feed back, 
but no established competencies or criteria upon which to conduct the evaluation existed. The education program had designed a 
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survey that measured the quality of its program, but opinion rather than student learning was the focal point of the assessment. 
Although the institution required students to take the Academic Profile or the CAAP tests as exit exams, the findings weren't shared 
with faculty members in a way that allowed them to make curriculum changes. More significantly, the team concluded, most of the 
current assessment plans did not provide an effective means of generating the data needed for sound decision making, including 
budgeting and planning. 



Finding Our Way 

Afterthe NCA team's visit, assessment committee members were determined on resolution of the assessment issue by the time the 
requested progress report was submitted in 2002. Thegroup reorganized as the student outcomes assessment committee (SOAC) and 
expanded its membership to include individuals who had responsibility forthe various components of the assessment process. NCA 
team members had noted in their report that administrative roles needed to be created to ensure consistent practice and monitoring. 
The new committee was composed of the vice president of educational services, the director of institutional planning and 
effectiveness, two assessment counselors, an institutional research statistician, and five faculty members from the AAS and AA 
transfer programs, respectively, who would serve three-year terms. 

In fall 1999 the committee established new goals, including 

o Determining what was currently done to assess student learning and development 
o Establishing and refining an approach to student outcome assessment 
o Guiding development of program assessment plans 
o Reviewing assessment processes on a regular basis 

o Providing appropriate training and educational opportunities for faculty and staff to facilitate the necessary awareness and 
understanding of key assessment issues and practices 

o Distributing assessment information to faculty, staff, administration, and community 

o Formalizing assessment programs to curricular budget and planning processes 

o Marketing the college, divisions, and faculty's assessment process 

o And, generating an awareness of the goals and student assessment throughout the college that extended beyond campus 
boundaries (State Fair Community College, 2002, p. 2) 

Before this, the faculty had significant administrative support for assessment initiatives. Money was approved for travel to various 
assessment conferences at which faculty networked with faculty from other institutions, many of whom were equally clueless about 
the direction an institutional plan should take. A mini-grant program drawing from Funding for Results money was established to 
encourage innovative projects for assessing student learning, but the projects related more to course-level than program-level 
assessment. ItwastheNCAmeetingin Chicago in 2000, however, that introduced committee members to the document that provided 
the road map: Cecilia Lopez's “Levels of Assessment.” The SFCC director of institutional effectiveness adapted Lopez's levels Into a 
notebook format, complete with a series of questionnaires intended to help program coordinators determine the focus fortheir plans 
and the most efficient way to conduct them. The notebook would also become the means for documenting data and the resulting 
changes. SOAC members were first oriented to the format, and they in turn served as mentors for program coordinators. As programs 
began to move through the various phases of assessment, coordinators identified missing lin ks in the process. Some learned of better 
testing methods, forinstance, but lacked the funds to implement these methods. To provide the necessary financial assistance during 
this period, the mini-grant program was refocused to limit funding to projects that directly related to program assessment ratherthan 
individual classroom projects. This Included support for the LS 2000 software system that has been implemented in the nursing 
program to provide integrated tests at the end of each content area. 

Although some AAS faculty are making more progress than others, all are following the same rules of the road. The structured levels 
of Lopez's plan not only helped unify the faculty's understanding of the process, but also simplified the institution's task of 
documenting It. To keep the programs on task, SOAC designed a three-tiered rubric to serve as an evaluation tool for the various 
program plans. The rubric was introduced to the program coordinators in fall 2001. Duringthe followingyear, all program coordinators 
were asked to submit their notebooks to a team of SOAC committee members who determined which assessment level the plan had 
achieved. These teams, in turn, presented their conclusions to the SOAC committee along with recommendations for improvement. 
In addition, the administration revised the program coordinator job descriptions as well as faculty evaluations to include student 
assessment. 
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In the meantime, the faculty responsible for general education program assessment were sitting in an intersection arguing about 
which direction to take. In fall 1999 a student outcomes assessment subcommittee revised the original general education assessment 
plan to include three studies: the Academic Profile entrance and exit exams, transfer data from Central Missouri State University and 
the University of Missouri in Columbia, and an internal study conducted by general education faculty. This subcommittee also 
recommended the development of a general education assessment committee with ten faculty members and one ex-officio member 
who would analyze the evidence collected to assess the program's five goals, which had been defined three years earlier. Several 
members ofthis committee, however, openly opposed suggestions for collectingdata-either because they didn't believe the method 
could be efficiently implemented in their discipline orthey believed nothing would be done with the data. In addition, members were 
concerned that the plan would beimpossible to implement at all ofthe off-campus sites where general education courses were taught. 
Another factorthatcomplicatedthe issue wasthe Missouri CoordinatingBoard for HigherEducation's(MCBHE) directive forallgeneral 
education core courses. It had recommended that all state institutions implement an assessment component because ofthe growing 
number of articulation disputes between community colleges and transfer institutions. The MCBHE's suggested goals and competen- 
cies were still in the drafting stage when the SFCC general education assessment committee began meetingto fine-tune its plan, and 
the group questioned the rationale of involving the faculty in an internal course assessment ofthe original goals when the institution 
must soon comply with the new state goals. 

Because little data had been collected to assess SFCC's general education program, committee members agreed, some grudgingly, 
that two courses for each ofthe five goals would be assessed each semester until the state goals were in place. The process would 
involve two members ofthe general education assessment committee and the course's lead instructor who would together examine 
evidence ofstudent performance from each class. Thissamplingofevidence would consistofworkfromten or more studentsand might 
include final exams, portfolios, essays or responses to essay questions, videotaped presentations, or lab worksheets. Consistent with 
the Academic Profile's means of reporting results, this evidence would be evaluated according to a three-tiered scale: The first tier 
suggests that students have no understanding ofthe goal's intended outcomes, the second that students have an acceptable 
understanding, and the third that they have an exceptional understanding. 

The first round of data was collected in spring 2000, but the findings were reported for only three ofthe five goals. No evidence was 
collected for one goal because the faculty misunderstood the type of data needed. Although evidence was collected for a second goal, 
the resultswere never interpreted. Furthermore, as the group had predicted, the process of gathering data from off-campus sites was 
frustrating. Adjunct faculty taught most ofthe courses selected for assessment at these off-campus sites, and for reasons including 
poor communication, much of these data were never collected. 

Fortunately, the general education program's external assessment methods appeared to be a more efficient means of generating 
evidence for analysis. The study of GPA and graduate rates of CMSU and UMC transfer students was a valued tool, although gaps 
existed in data collection and CMSU's reporting format differs from UMC's. The Academic Profile was more reliable. All students 
pursuing associate of arts degrees had been required to take the test as an exit exam since 1991. Because the Academic Profile 
measured the general education areas of math, writing, reading, natural and social sciences, and humanities, generating both 
nationally normed and total scores, the committee had concrete evidence for ten years ofstudent performance. Furthermore, the test 
was designed in a framework that could be meaningfully interpreted by many faculty members. The Academic Profile identifies th ree 
levels or tiers of skill for each general education area it tests. In fact, the descriptions forthese three skill areas directly correlate with 
the competency levels for each of the three courses in the program's math and writing sequence. The test also suggests that the 
mastery ofkeyreadingskillsattier levels I and II is directly related tothemasteryofcriticalthinkingskills. To provide more substantive 
data, all first-time degree-seeking students were asked to take the Academic Profile as an entrance exam as well in the fall 2000 
semester. Although the institution's assessment counselors have struggled the first two years to convince enough students to comply 
forthestudyto be significant, it is hoped thatthiscomponentwill provide more information in the future regarding how much students 
learn in the general education area while they are enrolled at SFCC. 

At a celebration of learning session in April 2001, the results ofthe Academic Profile were presented to the faculty as evidence that 
assessment is a valid exercise that reaps meaningful information. Interpretation ofthe data suggested that SFCC's students usually 
scored slightly above the national norms in all seven general education areas. Furthermore, when the fall 2000 entrance scores were 
compared with the long-range averages, the findings suggested that students made significant gains in all general education areas. 
However, an analysis ofthe performance level students actually reached at the point of graduation indicated room for improvement. 
While 37 to 51 percent achieve at the level 11 tier, only 10 to 14 percent ofthe graduates actually progress to level III. For this reason, 
faculty were encouraged to consider curriculum changes that might move more students into the level III skill area. In response, many 
accepted the Academic Profile as a credible tool for assessment, and some even agreed to consider curriculum changes to address 
the issue. Finally, afternearlyfiveyearsofresentmentandconfusion,theprocessofassessingthe institution's general education core 
courses seemed to be a valuable means of providing insight about what students were actually learning. 

In fact, some faculty were now convinced that the Academic Profile was the only assessment tool necessary. But most members ofthe 
general education assessmentcommittee agreed that becausethetest measures onlyskill levels and does not address theknowledge 
areas embraced by the general education core classes, other assessment methods must be in place. This continued to be a source 
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of contention forthe committee. By fall 2001, the state general education goals had been approved. They consisted of four skill goals 
and four knowledge goals, and the committee members decided that the current plan for collecting data shouid be abandoned so that 
thegroup could focus on integratingthe institution’s goals with the state’s. Although some members ofthe faculty resented thestate’s 
directive, in retrospect it was the road maptheAAfacuity needed. Today, the new pian is structured in a matrix format, listinggoals 
and their competencies and the courses that address them, and the original plan’s three-tiered evaluation method has been adapted, 
in lanuary 2003 the general education assessment committee chair met with all full-time and adjunct general education faculty and 
asked that a unifying paragraph be included in all general education course syllabi. Faculty were also asked to include the specific 
goals and competencies that each course addresses in all fall 2003 syllabi, a task that will be supervised by department chairs. Most 
remarkably, thejanuary 2003 meeting marked the first time thegeneral education faculty has met without disagreement in fouryears. 



Moving onto the Superhighway: An Institutional Culture of Assessment 

Looking back, three factors have played an integral role in helping SFCC faculty recognize the value of formal assessment. 

0 Communication. Assessment workshops and roundtables are a part of each fall and spring alLfaculty in’Services. These 
activities have included a best practices fair in August 2001, at which faculty discussed classroom assessment techniques 
(CATs) and, in January 2002 an evaluation ofthe status ofthe institutional assessment culture. A quarterly assessment 
newsletter has also been created; articles highlight the efforts of faculty and administration and provide documentation of 
results. This newsletteriscirculated at off-campus sitesand in thecommunity. And, since spring2oo2, a colorfulstudent flyer 
highlighting the purpose of course assessment is included in new student orientation packets. 

0 Faculty ownership. Although program coordinatorsand faculty are expected to workwithin the institution’s assessment plan 
framework, theydetermine what evidencewill be collected and thecriteriaforits evaluation. For instance, in the last two years, 
the English faculty has simplified the assessment ofthe English composition sequence significantly. Although the NCAteam 
noted that the English faculty had established competencies, developed a rubric, and made textbook and other curriculum 
changes based on the data they had collected, the exercise was a tedious chore carried out each August that most ofthe 
English facultydreaded. Now, the faculty collectively evaluates only a random samplingof source-based papers gathered from 
allfalland spring sections of English Composition land II. All full-time and adjunct facultyfrom all campus sites meet tograde 
six students’ work, their Composition I essay matched with the paper they completed in Composition II. In turn, each faculty 
membergrades five additional sets of papers with a partner. Notonly isthethree-houractivityan effective means ofgathering 
data, but it provides an opportunity for meaningful discussion about student performance and builds faculty unity in regard 
to grading standards. A Funding for Results mini-grant provides meals and a stipend for adjuncts. In August 2002, TECH-PREP 
provided stipends to include twelve area high schools in the process as well. 

0 Documentation ofthe results. Most importantly, a number of changes have been made as a direct result of assessment in 
the institution. 

o The results of and ASEend ofthe program test for the automotive technology program indicated that the area of 
electronics andtransmissions neededto be strengthened, andthe program coordinatorconstructedmock-upsand 
trainingaidstoimprovestudent performance. Datawill be collected from futureexitteststodeterminewhetherthat 
change has contributed to improvement. 

o As a result of data input from its advisory council, the industrial management program has replaced the 
requirement for microeconomics with an elective, and a new three-credit-hour quality management and control 
course has combined two previous courses that addressed the same objectives. 

o A one-credit-hour critical thinking component has been added to the two-hour developmental reading course to 
improve the critical thinking scores on the Academic Profile. 

o Asa result of a fall 2000 pilot program for college algebra that indicated that students’ performance improved with 
five hours of instructional contact per week, the course has been expanded from three to four credit hours as of fall 
2003. Following in suit, and as a result of an evaluation of final exam scores, basic math and elementary algebra 
classes have been expanded to a five contact hour class as of fall 2002. Students pay for three credit hours but 
participate in five contact hours a week. 

o The business management program has integrated more technology into many of its required courses as a result 
of the evaluations students received during their business and industry internships. 

o Data collected from the NCLEX-RN pass rates in the nursing program supported implementingthe LS 2000 software 
program to test student’s knowledge at the conclusion of each content area. In addition, mentors were assigned 
to students with low scores in the integrated nursingcourse, which resulted in all students’ successful completion 
ofthe course. 
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So, are we there yet? Although manyfaculty members would agree that assessment is finally an accepted and respected component 
of State Fair Community College’s institutional culture, this adventure will have many sequels. Changes in administration and state 
as well as federal requirements for reporting student outcomes will create frequent challenges. As the college continues to add 
programs, expand its Web and dual-enrollment offerings, and accommodate the constantly changing needs of the American 
workplace, theassessment process will more than likely change as well, servingan integral role in theguarantee of academic integrity, 
far into the next century. 
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The Road Not Taken: An Assessment Program 

Where Everyone Wins 

Karen jones, Michael L Banks, Jacqueline Gray 



Introduction 

“Two roads diverged...long I stood and looked down one as far as I could." Robert Frost’s The Road Not Taken about choices and their 
consequences accurately describes the dilemma ^cing most schools as they try to figure out what must be done with assessment. 
Looking“asfaras I could” often invo Ives attendingand sending people to the many assessment conferences that have sprung up over 
the past fifteen or twenty years. On most campuses, the impetus for looking down the road comes less from a sense of institutional 
need to know how students are doing than from a response to demands from state agencies and regional accreditation groups. In his 
keynote address at the 2002 MHE Assessment Conference, Peter Ewell noted “the fact that it [assessment] almost always begins at 
the behest of somebody else creates a pervasive feeling on the part of faculty of being 'done to’ rather than doing” (2002, p. 3). St. 
Charles Community College’s (SCC) early attempts at assessment illustrate that reality quite clearly. 

In the mid- to late 1980s, the college, led by administrators, created a committee that reviewed some standardized tests and plotted 
ways to get students to take them. The path to be taken seemed smooth, flat, and obvious. Choose the test, make taking it a graduation 
requirement, charge students a fee to cover the cost, and report to all interested parties that SCC students had above average scores. 
And so the assessment car, chauffeur-driven by the testing company, pulled out of the station carrying satisfied administrators along 
a smooth path. 

In retrospect, the problems inherent in that original plan are glaringly obvious. Students were not charmed by the idea of using their 
time and money on another test they perceived as irrelevant. In addition to complaining, they completed tests in record time and 
escaped. The results reported by standardized testing companies still indicated above state and national average results. Academic 
administrators created charts and graphs. Faculty members ignored the entire thing; after all it was an administrative problem. And, 
for a time, the demands from outside constituencies were quelled. The road had bumps and potholes, but adequate padding kept 
administrators comfortably dozing in the back seat. 

Of course, some things changed relativelyquickly. Students were bribed,notcharged,toactually show upandtaketesting“seriously." 
Testing supervisors distributed bookstore discounts, T-shirts, and food vouchers. But things rolled relatively smoothly until outside 
evaluators arrived for an accreditation review. Their evaluation of our assessment process and reports brought the assessment car 
to a halt so abrupt that the administrators in the car suffered whiplash. 



Creating a New Assessment Plan 

After the usual shock, puzzlement, and finger pointing, SCC administrators, a courageous lot, regrouped and rethought. The results 
of that decision to begin again created a process for assessment that engaged faculty and has already resulted in curricular changes 
and judgments about program content. Beginning in fall 1998, assessment was turned over to faculty. A faculty task force with 
members from all of the academic divisions of the college began by engaging the entire faculty in a dialogue. The campus curriculum 
committee had just reviewed thegeneral education requirements fortheAAtransferdegree. As partofthatreview,they had developed 
a goal statement for general education and for each categoryofgeneraleducation. DuringAYi998-i999, faculty began reviewingand 
revising the general education assessment process using the general education philosophy and goals recommended by the 
curriculum committee in spring 1998. 

The faculty assessment coordinator and division assessment leaders (all faculty members) worked with faculty in discipline areas to 
develop general education objectives for each of the six subcategories within the general education requirements. Faculty members 
reviewed the specific course objectives, and by the end of the academic year most had identified objectives congruent with general 
education goals. Faculty in each subcategory created instruments and processes to be used in 1999-2000 to measure student 
outcomes. 
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The General Education Assessment Plan 

Because the faculty decided to base assessment on locally developed goals and objectives, each of the areas represented in general 
education created a locally developed assessment project with a goal of improving coherence in curricula and student learning. 
Assessment projects were developed in writing, speech, humanities, social science, mathematics, and computer literacy. 

0 Writing assessment. Since 1992, English faculty members have used a holistically scored entrance essay to place students 
in one of three beginningwriting classes— two developmental and one college level. The process was expanded by requiring 
a department final exam of all enrolled in the developmental courses. Observations from that scoring process and a review 
of pass-fail rates led English faculty to revise placement criteria and curricula in both developmental writing classes. Those 
revisions improved the success rate of students in subsequent classes. 

Beginning in fall 1999, all English Composition 11 students write a required essay based on a packet of readings distributed 
one week priorto final exams. Students receive a grade from their instructor. A random sample of student writing is evaluated 
independently by the English department reading team, using an analytic rubric keyed to course objectives. Scores are 
reported by objective to all writing teachers, and strengths and weaknesses are discussed. Several units in composition were 
strengthened, and a more structured direction sheet was developed for use in student final exam preparation. 

0 Oralcommunication. Oral communication combines common final exam test items with a reviewofa random sample oftaped 
speeches. The department routinely taped student speeches for students. Students were asked to study their own 
performances as recorded and provide self-assessments. As faculty members reviewed a broad sample of student tapes, they 
discovered both equipment and training needs within the department. Part-time faculty have become a part of the 
assessment process as information on student performance has begun to reshape curricular emphasis within the required 
class. 

0 Math and science assessment. On all final examinations, math and science faculty members use a set of common questions 
keyed to specific core objectives. Data are aggregated and reported to faculty. Recently, both departments have begun to 
correlate student scores with course grades. Math faculty members continue to struggle with problem selection for inclusion 
in final exams. Correlations between student scores on department-generated problems and student course grades show 
little relationship at present. The department is reviewing both core course objectives and testing methodology.Science 
faculty members developed a set of common questions that focused on student ability to apply scientific principles. Afterthe 
first results were analyzed, it becameclearthat certain objectives were better measured in the lab portion ofthe course. When 
assessment began, all science courses met the general education science requirement. Based on the scores of students 
enrolled in science courses without labs, the department recommended that only courses with linked labs be included in 
general education. 

0 Social science and humanities assessment. These general education areas represented the greatest challenge, since both 
areas encompassed many disciplines. Both groups decided to begin collecting artifacts from students enrolled in courses 
meetinggeneral education requirements and evaluate them based on rubrics developed by faculty in each area. In fall 2000, 
based on the results ofthe pilot project, faculty members adjusted the rubrics. They also agreed to ensure that at least one 
written assignment would be focused on general education objectives. Because the process provided only limited data, 
faculty members agreed to accumulate a minimum of four semesters of data prior to major curricular review. 

0 Computer literacy. Courses in computer science were required as a part ofthe general education core. Students were able 
to test out ofthis requirement by takingan examination to demonstrate theircomputerskills. The original assessment project 
developed for general education had no connection to that test-out exam. The difficulty in defining computer literacy and 
connecting that definition to the courses offered to meet the requirement resulted in two significant changes. Required 
computer courses were eliminated from the general education core. However, completion of specific courses continues to be 
a graduation requirement. In addition, computer science faculty have developed an entirely newtest-out instrument and use 
it to assess the success ofthe course. 

0 Career-technical program assessment. Until 1999, career and technical program assessment was confined to licensure 
exams required by programs like nursing and to employment and satisfaction surveys. While those exams and surveys still 
provide information, other measurement processes have been added. General education for career-technical programs is 
often program-specific. In addition, faculty wanted to have a better idea of what specific skills students were acquiring and 
retaining. 

Theirsolution wastocreatea capstonecoursethat would incorporate criterion-referenced standardized testingand port folios 
to document student accomplishments. The majority of programs adopted a minimum of two elements from ACT-Work Keys 
as the general education component. This test allowed individual programs to focus on skills they believed most essential to 
their students and to establish targeted skill levels. Students worked individually with the capstone instructor to create 







82 



BEST COPY AVAILABLE 



Chapter 2, Assessment Processes /zz 



portfolios and other materials appropriate to their program. Students have used those portfolios in job interviews to 
demonstrate their qualifications to prospective employers. Faculty have used one-on-one meeting times to assist students 
in their search for jobs after graduation. 

Several programs have modified course requirements and created or deleted courses as a result of what has been learned 
from the capstone process. As general education requirements for career-technical programs undergo review, the standard- 
ized testing modules will provide information for any changes. 



What We’ve Learned 

It’s possible to involve the majority of faculty in the assessment process. The faculty at SCC wanted nothing to do with assessment 
in 1999. If in 2003, someone suggested eliminating our assessment process, faculty would march on the president’s office. They own 
it, they run it, and they have come to believe that the process of assessment improves their teaching and the education of their 
students. 

It’s impossible to have assessment without bloodshed-metaphorically, at least. On most campuses, the curriculum committee has 
the greatest potential to create campus-wide debate among faculty. At our campus, assessment “discussions” can be added. The 
contemplated shift in general education assessment to a capstone course had various groups insisting that “their” assessment 
projects be allowed to continue alongside the proposed capstone experience. 

It’s possible to create assessment projects that provide information about student performance relevant to the particular course 
structure of an individual college. Many colleges have done so and have shared information aboutthose plans at assessment meeting 
across the country. 

It’s impossible to create a “one-size-fits-all” assessment plan that transfers to any college. Many elements in the SCC plan were 
inspired by projects reported by other colleges. But ultimately, as the process continued, our assessment responded to the demands 
of faculty for information immediately relevant to their students and their courses. 

It’s impossible to be “finished.” Every question “answered” prompts another as yet “unanswered.” 

At SCC, assessment is messy, contentious, and faculty-driven. Administrators provide assistance and support. Students benefit. All 
of us now travel together on foot- the limousine sailed down the well-traveled road, and we took the other path. 
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“Now Is the Summer of Our Discontent”: 
Assessment Revisited 

Lisa Brandom 



The assessment program at John Brown University, which is a small comprehensive university on the border of northwest Arkansas 
and Oklahoma, is now entering into its second season of implementation, its summer. The springtime of the program occurred in the 
mid to late 1980s with a flowering flourish as the university’s vice president of academic affairs led the initiative. He appointed a one- 
quarter-time associate dean for institutional research who also served as registrar and self-study coordinator to prepare for an 
upcoming North Central Association visit. The institution then went into an intense mode of preparing the assessment soil (by 
attending regional meetings to learn more about assessment of student learning). The next step involved fertilizing the soil (by 
organizing frequent meetings with the academic departments to ensure understanding of the directive), and, finally, working to 
produce healthy plants (by establishingappropriateassessmenttools,evaluatingthe results, and improvingstudentlearningthrough 
curriculum and administrative changes). 

The university is now some fifteen years into its assessment program and has recently completed an evaluation of its current status. 
The program is almost routine, but is the summer season of its growth cycle the right time to become complacent and simply wait for 
the harvest? What can a university do to keep the focus and edge on the improvement of student learning? As a small university, how 
can it keep the administrative costs of the program down with the influxof many additional competing financial needs? How can it 
continue to motivate faculty and students to take ownership of the program? Small universities such as ours do not have to become 
discontented or disheartened in the summertime of their programs, and healthy plants raised in the assessment climate do not have 
to droop and become a wasteland because of drought and lack of care. With some revitalized ideas, the assessment programs can 
move into autumn with confidence. 



What can a small university do to keep the focus and edge on the improvement of student 
learning? 

One ofthe ideas John Brown University has initiated in recent years is the creation ofa yearly assessment summit on campus. For the 
pastthreeyears, on a cold autumn eveningin November, various campus constituencies come together toconsider assessment. They 
spend several hours together (rewarded with a Mexican dinner midway through the evening), examining the state ofthe assessment 
program and setting new goals for the current year. 

Those in attendance include academic divisional chairs, student life representatives, the director of institutional research and 
assessment, enrollment and financial aid representatives, student leaders, and others. Prior to the meeting, the office of academic 
affairs circulates amongthe attendees a bound document containing all kinds of data for analysis and conclusions. Examples of these 
data include the university's priority goals, results of surveys such as the Cooperative Institutional Research Project (CIRP), the Student 
Satisfaction Inventory, and the NationalSurveyof Student Engagement, aswell as data on enrollment, retention, and graduation rates. 
So that the participants are not overwhelmed with too much data, each section ofthe document is p receded with a one-page executive 
summary ofthe results. 

The meetingofficially begins with an analysis of the specific action goals that were developed by the participants at the previous yearns 
summit. The action goals are three assessment priorities for immediate focus. The three identified in 2000 were (1) retention and 
graduation rates, (2) academic quality and faculty/development scholarly activity, and (3) a need to reviewthe university's assessment 
surveys and their use. Each participant completes a brief open-ended question related to the three action goals and responds to a 
series of statements related to each goal. The large group then discusses how well the university has accomplished the action goals 
from the previous year. It is then time to develop new action goals. The participants divide into small groups and use flip charts to 
develop three action goals for the new academic year. Each small group shares its suggestions and discusses them, and the large 
group chooses the final three for the university's attention in the immediate future. 

Essentially, the summit provides camaraderie and a sense of shared ownership in the assessment program. As one of the nine 
principles of good practice for assessing student learning, the American Association of Higher Education recently stated: 
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Assessment is a goal-oriented process. It entails comparing educational performance with educational purposes and expecta- 
tions— those derived from the institution's mission, from faculty intentions in program and course design, and from knowledge 
of students’ own goals. (Astin et al., 2003) 

The assessment summit gives the university a cost-efficient way to share assessment results and cooperatively develop new goals in 
an ongoing manner. Astin et al.(2oo3) also state as a principle that “Assessment works best when it is ongoing not episodic.” The 
assessment summit provides a convenient vehicle for annual evaluation of its program. 



How can we keep the administrative costs of the program down with the influx of many 
additional competing financial needs? 

For small universities such as John Brown University, one of the key questions that has hovered like a bee in summertime is the cost 
of maintaining a quality assessment program. Many universities face uncertain finances in light of falling stock market prices, 
terrorism and war, and high unemployment levels. Student retention may fall; tuition is sure to rise; and enrollments may decline. 
More than ever, we must confront the economic realities. 

Much research has been conducted on why assessment programs have not had more success in the past fifteen or so years. A recent 
search on the Internet for articles related to the words “assessment of student learning,” resulted in 1,590,000 articles or references 
to the subject. According to just one of these articles, “Improving the Assessment of Student Learning” published in Teaching 
Sociology {\Ne\ss et al., 2002), one problem universities must confront is that “doing good assessment does require time. Faculty 
members will inevitably resent having any time-significant task imposed on their workload without some corresponding reduction in 
other responsibilities” (p. 68). Other than giving faculty members released time, a costly endeavor, how can a university encourage 
them to stayinvolved in assessment? An answerfor small universities such as ours is to “showthem the money,”toborrowa line from 
the f\\m Jerry McGuire. In this case, however, the “money” is not monetary. It is in showingthe faculty the intrinsic reward-the positive 
results of systematic and continuous improvement and the benefit to faculty members’ own goals and objectives. 

An example of how this works is in the English department of John Brown University. Each three years, faculty members within the 
department work to develop a three-year plan for assessment of its majors. These objectives, and the accompanying results of student 
assessment, are reviewed annually. The department employs the usual standardized tests as assessment tools for its majors, such 
as the Praxis test in education and the Major Field Achievement Test. It uses multiple addition tools as well, such as students’ writing, 
their presentations at professional conferences, and videotaped exit interviews, as evidence of whether the objectives have been 
accomplished each year. In the past it used longitudinal methods such as tracing students’ growth from their freshman through senior 
years. 

0 

When it comes time to review the objectives and results, the department has found that the summer provides an excellent time for 
a day-long retreat to accomplish these evaluations. The fellowship and the realization that shared goals ofthe department are indeed 
being met override any feelings of serious entrenchment upon faculty members’ time. Other departments also use short retreats for 
the same purpose. No one person is responsible for departmental assessment: it is cost-efficient: and it is a shared responsibility. 



How can we continue to motivate students and faculty to take ownership in the program? 



Peter Ewell (1997), in his work on assessment of student learning, has concluded that many institutions do not possess a clear 
understanding of collegiate learning and that assessment is often done in a piecemeal fashion within the institution. Angelo (1999, 
pp. 3-4) further states that “most assessment efforts have resulted in little learning improvement because they have been 
implemented without a clear vision of what ‘higher’ or ‘deeper’ learning is and without an understanding of how assessment can 
promote such learning.” It would seem that a possible answer to these dilemmas is the development of an ongoing, unified, across- 
the-campus plan. John Brown University’s plan is three-pronged. One part involves the administration-led initiative ofthe assessment 
summit forcommunication of factual data relatingto students’ life goals and setting ofthe faculty’s priority goals for assessment. The 
second involves the staff-led initiative focusing on collecting, analyzing, and sharing relevant data on the core curriculum. The third 
involves the faculty-up ownership of departmental objectives, the yearly examination ofthe results of assessment data, and the 
necessary changes in the curriculum to improve student learning, thereby achieving the closed loop that is vital in assessment. All 
three prongs ofthe program work simultaneously and systematically in three-year growth cycles. All constituencies work together 
collaboratively to create a network in which no one person or group feels pressure to produce without being sufficiently rewarded 
through released time or extra incentive pay. 

Since John Brown University is just in the summer of its assessment program, it can only imagine the future challenges ofthe fall and 
winter. Perhaps a challenge will be to develop more outside roots in the surrounding community, making more public and visible the 
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positive accomplishments of its alumni who represent the harvest of the university’s assessment efforts. Perhaps that message needs 
to be communicated externally so that the university does not have to make continued apologies for being “the best kept secret in 
northwest Arkansas.” Another future direction of assessment will perhaps be to network more with other universities in planning, 
sharing, and communicating innovative ideas in assessment that are emerging each day. The use of e-mail and the Internet is another 
cost-efficient way to share this information. Perhaps the future direction will be to encourage more scholarship within the university 
that pays more attention to the multiple ways in which teachers teach and students learn. 

Small universities struggle with issues of staying on the cutting edge of assessment, working with small budgets set aside for the 
program and continually involving students and faculty without burdening them. Employing some simple, almost cost-free summits 
and day retreats can perhaps provide the continuing discussion and planning times needed to accomplish the objectives. The phrase 
“the summer of our discontent” is simply a reminder that contentment is not necessarily a positive. Institutions such as John Brown 
University must remain discontented as they seek ways to move to the next cycle ofgrowth in academic excellence and effectiveness. 
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A Balancing Act; Engaging Faculty 
and Admisnislralion in 

University-Wide Assessment 

Donald Carpenter and Badih Jawad 



This presentation seeks to engage the audience in constructing a model for a university assessment program. The model developed 
by the audience will then be compared to the program implemented at Lawrence Technological University (LTU), where assessment 
of student learning is sustainable and is part of the university culture. The ultimate goal of the case study is for participants to adapt 
and implement the model for use at their home institutions. 

The university-wide assessment committee was created in fall 2001 and consists often faculty representatives (one from each 
academic department on campus). In addition, the committee has two ex-officio members, the associate provost and the coordinator 
of institutional research. The associate provost serves as a consultant to the group and a liaison between the committee and the 
provost's office. The coordinator of institutional research provides data services and general assistance. The principal presenter, 
Donald Carpenter, Ph.D.,isanassistantprofessorof civil engineeringand an active member ofthe committee. Theco-presenter, Badih 
Jawad, Ph.D., is an associate professor of mechanical engineering and the director of assessment for the university. The director of 
assessment serves as the chair of the assessment committee, and his responsibility is to make assessment of student learning part 
ofthe university culture. 

The assessment committee, which meets biweekly, serves as a channel of communication between university departments and a 
forum forfaculty suggestions on assessment issues. In addition to attendingthe regular meetings, the director of assessment and the 
ex-officio members meet once a semester with each faculty representative and their department chair to review departmental plans 
and assessment activities and discuss actions the department is planning for the upcoming semester and beyond. At the time of 
committee formation, the various departments on campus were at extremely different levels of tracking student learning. Some 
departments had never attempted anyassessment tracking, while others had developed rathercomplex methods of assessment. This 
variety turned out to be a great situation for the committee because it allowed for an opportunity to combine the fresh ideas of 
newcomers with the experienced ideas of those departments that had been tracking student assessment for years. Numerous 
methods of assessment were introduced, includingthe concepts of direct and indirect measurements. An early study was conducted 
with the committee and the faculty in their departments to determine the level of implementation in the departments. Each 
department reviewed areas of culture, responsibility, support, and efficacy of assessment to give the committee an overview ofthe 
implementation levels throughout campus. While some departments were at the lowest level of activity; others were at a more 
advanced level. 



One ofthe first tasks ofthe committee was to determine a focused plan for the year. It was agreed that assessment would initially be 
tracked at two levels— the university-wide level and the department level. The university-wide component followed Lawrence 
Technological University’s educational goals ofcommunication, development ofanalytical and critical thinking skills, teamwork, and 
leadership. The department level of assessment would be focused on goals and objectives of each department, which, in some cases, 
were aligned with accreditation boards. 



The committee started with the LTU educational goals that were already in place. The educational goals consist of six main general 
educational requirements as well as seventeen specific skill areas in which graduates are to demonstrate. The committee recognized 
very quickly that it needed to narrow the scope of work and chose to concentrate on four specific skill areas: 




o Effective verbal and written communication skills 
o Critical thinking and problem-solving skills 
o Teamwork 
o Leadership 
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Initially, assessment ofwritten communication skills was chosen as ourfirst university-wide assessment task. The committee realized 
that, in the case of Lawrence Tech students, writing should be assessed at two levels-as part of the general education requirements 
and core curriculum at the sophomore level, and as professional and technical writing at the end of the junior or senior year. The 
assessment of writing in conjunction with general education is performed by the department of humanities, social sciences and 
communication. The assessment is integrated in the writing, literature, history, and philosophy courses taught as part of the core 
curriculum. Random samples of student workarecollected forthe various coursesatthe freshmen and sophomore level and evaluated 
by a faculty team. 

The majority of the committee's work in the first year was determining a method to review upper class students' professional and 
technical writing. Various methods of writing assessment were reviewed. Ultimately, the committee agreed to have student writing 
reviewed by a small team of faculty members from across the departments who are known to have exceptional writing skills 
themselves. This team will be responsible for reviewing a variety of student professional papers to determine the writing level of our 
students. Professional papers from students at the end of their junior and senior years, representing every undergraduate major on 
campus, will be collected and reviewed. Wanting to provide a general framework for the team to follow, the committee developed a 
writing rubric to establish a standard of grading for writing and to show the expectations of each grade. 

In addition to the efforts in assessing written communication, each department worked toward departmental specific goals with the 
guidance of the respective committee member. The departments were asked to develop their goals and have them documented by the 
end of the academic year. The departments showed great initiative in their work, and a variety of assessment strategies were 
developed, ranging from student surveys and exit interviews to pre-testing and post-testing in specific courses. Another accomplish- 
ment of the assessment committee was the development of assessment week each semester. Lawrence Tech's assessment structure 
includes one week persemesterthat culminates with an assessment day full of scheduled events. The fall assessment weekis focused 
on the evaluation of assessment results from the previous year and the actions to follow. During the fall assessment day, the faculty 
participated in program retreats and in a faculty symposium to share assessment practices and results. In the spring semester, the 
assessment week will be focused on the collection of student data. This is a focused time frame for faculty to collect a variety of data 
in the classroom as well as an opportunity for campus-wide surveys to be distributed. 

All assessment committee members attended at least one conference related to assessment during the first year. Most of the 
committee went to Chicago in March 2002 to attend the shared conference of AAHE and the Higher Learning Commission, "Changing 
Institutional Priorities.” During this workshop our committee had the opportunity to hear from other institutions and to learn many 
ways to improve our methods of assessment. In addition, the conference allowed ample opportunity to grow together as a team and 
to develop a capstone project. This capstone project gave us time to focus on goals, strategies, and indicators of successful student 
learningassessment. The conference was time well spent because it furnished ideas and suggestions from otheruniversities that have 
been very successful with their assessment strategies. Some highlighted suggestions included sharing assessment methods with 
students at orientation and other student gatherings, starting with small plans and building upon them, and learning that each 
assessment method has strengths and weaknesses. 

Although the first year of our assessment committee went well, the group still faced challenges. One of the first noticeable 
discrepancies came from the fact that Lawrence Tech is made up of several professional colleges with different expectations from their 
professional accreditation agencies (engineering, architecture, business management, and departments without professional 
accreditation). As a starting point, the committee prepared a chart showing the various agencies and the languages defined by those 
agencies (the Higher Learning Commission, FIDER, NAAB, ABET, lACBE, and ACBSP). It was immediately evident that we were all 
speaking different languages when trying to identify common goals among these diverse departments. 

The committee members were dedicated in their pursuit of assessment of student learning. Every department made an effort to try 
various methods of assessment and to learn from the ideas of others on the committee. In addition to the dedicated work of the 
committee members, several positive factors led to our success. The administration provided a separate annual assessment budget 
to support assessment activities. The budget is used for a variety of items such as conference travel, educating faculty about 
assessment, consultant fees, and clerical or technical assistance for special projects. Technological tools, such as Blackboard, are 
available on campus to enhance communication between various constituents of the process. There was broad institutional support 
of the work of the assessment committee, as well as support of the assessment plan. Support came from all levels on campus. The 
president, provost, deans, and department chairs continually showed their support and contributed to the success of this committee. 
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Broadening the Discussion: 

Using the Academic Assessment Process to 
Address a Variety of Institutional Priorities 

Jacqueline Johnson and Robert Ferguson 



Background 

For the pastthreeyears the faculty at Buena Vista University (BVU) have been engaged in a process of reviewing our general education 
program and each of our majors. In May 2002 the faculty gave final approval to many of the major components of a new curriculum, 
central to which are eleven BVU objectives that are to be integrated throughout all facets of a student’s education. During the summer 
following passage of the program, the assessment committee began working to develop a proposal for a new academic assessment 
plan to fit the new curriculum. 

The assessment plan that has been developed envisions that the eleven BVU objectives will be addressed on a three-year rotating 
basis, beginning with three of the objectives this year. The basic strategy for data collection will be to identify existing senior-level 
courses in each major that contain assignments relevant to the objectives. Copies of the work from the students in those courses will 
beforwarded by theinstructors to the office of institutional research, where they will be stripped of identifying information and pooled 
with work from other courses. A scientific sampling process will then reduce the student artifacts to a manageable number, and a 
faculty committee will assess the work using rubrics developed earlier in the year by faculty volunteers. The assessment committee 
will review the results, make them available to the university community, and use the annual beginning-of-the-year faculty workshop 
as a forum for initiating campus-wide discussions of the results and their implications. 



Relationship to Other Institutional Issues 

The central purpose of the assessment committee, of course, has been to develop and implement a plan forthe collection of data that 
would allow us to gauge student academic success. What we did not anticipate was that the process would yield significant benefits 
even before any formal data were collected. In particular, three areas of concern have long been identifiable in the Buena Vista 
University campus culture: 

1. Faculty attitudes regarding academic assessment have often ranged from apathy to distrust. 

2. Several segments of our curriculum, including components of general education, have operated rather independently of one 
another, often with minimal coordination or little sense of how the components interact to create an overall academic 
experience for our students. 

3. Communication between the faculty on the main campus and the faculty and staff members who operate our baccalaureate 
degree-completion programs in partnership with community colleges throughout the state of Iowa has not always been 
adequate. 

The process of developing and implementing a new assessment program has led to discussions among faculty from all segments of 
the university. This has provided a mechanism for addressingthese three areas of concern, all of which have already shown significant 
improvement in spite of the fact that we are only just beginning to collect formal assessment data. Each area and its relationship to 
the assessment plan will be discussed in turn. 

□ Faculty Skepticism 

As has been the case at some other institutions, BVU faculty members have not been quick to embrace the assessment process. 
Some have seen assessment as Irrelevant and hoped that it would go away, while others were more openly distrustful. Thus, as 
the assessment committee began its work, all members were very aware that acceptance of the plan would require it to be seen 
as both relevant and non-threatening. The committee therefore identified three principles to guide its deliberations. 
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1. Good assessment is designed as an integral part of the academic process, not as an artificial or intrusive add-on. 

2. The intent of the process is to assess overall programs and BVU objectives, not individual courses, instructors, or 
students. 

3. The new plan would represent a long-term commitment to thoughtful, high-quality assessment of student learning at 
Buena Vista University. 

These principles are reflected throughout the plan. In particular, they led to the decision to draw primary assessment data from 
existingassignments in courses across thecurriculum. This approach does not requiresignificantadditionalworkforinstructors, 
and it ties assessment to the day-to-day academic life of the institution. Also, we elected to strip identifying information from the 
student work and then aggregate it across courses and majors, thus getting an overall profile of performance from students as 
they prepare to exit the institution without targeting individual instructors or courses. 

One significant measure of the acceptance of this approach came during the faculty meeting at which the plan was presented 
priorto submission to the governance process. That meeting sawwidespread discussion (as opposed to apathy), capped by the 
comment from one long-term skeptic that “I think we can do this!” More concrete evidence for the plan’s acceptance can be seen 
in the fact that nearly thirty people volunteered to participate in a weekend workshop to develop rubrics for the first three BVU 
objectives, and we have met little or no resistance when soliciting faculty participation to identify assignments and gather data. 

□ Minimal Coordination of Curricular Components 

The BVU objectives are intended to be integrated throughout all aspects of a student’s experience, including general education, the 
majors, and, to some extent, the extracurricular programs. The simple act ofidentifyingthese objectives has gone a longway by itself 
toward identifying points of connection and commonality among otherwise diverse programs. Equallyimportant, however, has been 
the development of assessment plans for the objectives. For example, the academic affairs committee of the faculty senate 
has asked all instructors offering courses within general education to submit a list of the specific BVU objectives addressed 
by the course, along with an indication of how those objectives will be assessed during the semester. This requirement, in 
turn, has led to several significant discussions about the interrelationships among courses. This may seem like a small step 
for some institutions, but to have it occur widely across divisional boundaries is a non-trivial accomplishment on our campus. 

□ Integration of Programs Between the Main Campus and the Centers 

For more than twenty-five years, Buena Vista has provided community college graduates across Iowa with the opportunity to 
complete their undergraduate degrees by taking two years ofadditional coursework at our centers, most ofwhich are located on 
the community college campuses. All students, regardless of whether they are enrolled at the main campus or at a center, are 
expected to complete essentially the same requirements, and they receive the same degrees. Academic deans and faculty from 
the main campus design the programs and approve the adjunct instructors at the centers. Nonetheless, coordinating the 
operation of the programs and courses among the various locations has sometimes been difficult. Developing the new 
assessment plan has proven to be an ideal mechanism for increasingthe dialogue and degree of coordination amongthe faculty 
and staff who actually deliver the programs at all of those sites. Several examples can be cited. Rrst, the dean of centers is a full 
memberoftheassessmentcommittee, so her perspectiveand concerns have been an integral partofthe overall discussion. Also, 
the chair of the assessment committee and the dean of centers have used this as an opportunity to travel around the state to 
discuss the assessment plan and its implementation with center faculty and staff. Finally, several center faculty and staff 
members participated alongside main campus faculty in a recent weekend workshop during which they developed the rubrics 
to be applied to the student work. Academic assessment is thus becoming one important mechanism driving increased 
discussion and coordination amongthe centers and main campus programs. 

Conclusions 

There is no question that assessment of student learning is a vital component of any academic program in its own right. Students, 
faculty, administrators, and other stakeholders need, and deserve, to know how well curricular goals are being accomplished, and 
curricular improvement hinges on good assessment data. In addition, Buena Vista University has discovered that an appropriately 
designed assessment plan can be of real value in addressing wider institutional issues. Academic assessment holds the potential to 
become a focal point around which many constituencies can converge and thereby find common ground. Articulation, integration and 
intentionality are significant characteristics in any good curriculum, and assessment can be the key organizing mechanism to further 
all of these goals. 
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Living Beyond Our Means: 
Analysis of Standardized Test Data 

John C. Simonson and George E. Smith 



Standardized tests have long played a significant role in assessment strategies at most colleges and universities. As part of the 
assessment landscape, the results of such tests have been used in institutional admissions procedures, as internal benchmarks for 
measuring added value, as input for curricular and administrative decision making, and for both internal and external comparisons 
of students* abilities in mathematics, writing, critical thinking, and other areas. 

The purpose of this paper is to provide insight into strategies for analyzing the results of such standardized tests. Based on activities 
at the University of Wisconsin-Platteville (UW-P) over the past decade, the paper will suggest methods of data analysis beyond the 
simple institutional averages often provided with standardized test results. 



Brief History of Assessment on Campus 

The University of Wisconsin-Platteville is one of thirteen four-year campuses within the University of Wisconsin System. Among the 
institution's assessment activities of the past five or six years have been development and testing of a home-grown instrument in 
mathematics, portfolio development in writing, creation of a campus “assessment journal,” and interdisciplinary assessment 
discussions across the various areas of the liberal arts curriculum. 

Program-specific assessment activities predated campus-wide efforts. For example, UW-P*s Department of Economics studied 
students* critical thin king skills in a series of pedagogical experiments in 1987. In the mid-1980s, the College of Engineering participated 
in a UW System-funded pilot assessment initiative that provided a foundation for developing program-specific assessments. 

In the early 1990s, UW-Platteville began to participate in UW System-wide assessment of writing and mathematics skills. In addition, 
UW-P also studied students* critical thinking skills. UW System administration mandated use ofACTCAAP for this assessment. Prior 
to this directive, UW-Platteville had considered other standardized instruments for conducting its own local assessment. 

The authorswereamongthe faculty and administrators concerned about such systematic assessment of UW-Platteville students. One 
result of this concern was the development of an institutional assessment plan (1995) that codified the university*s assessment 
philosophy, goals, strategies, policies, and procedures intoa single document. This plan collected input on assessment activities from 
campus academic and administrative units. 

In addition, this plan created an assessment oversight committee (AOC) charged with the responsibility of coordinating campus 
assessment activities and serving as a resource for academic and administrative units interested in particular assessment projects. 
The plan also articulated campus proceduresforcertain assessment activities (e.g.,biennialadministration ofACTCAAP). Finally, the 
plan promoted the development of home-grown assessment instruments, supported in part by the establishment of an assessment 
activity grant program to fund such activities. 

Assessment activities were divided into three general areas: competencies, liberal arts, and program assessments. A fourth 
component-student life-was added when UW-Platteville updated its campus assessment plan in 2002. 



Administration of ACT CAAP 




In part, the UW System decision to administer ACT CAAP at all four-year institutions prompted UW-Platteville to focus on specific 
proceduresfor managingthis effort locally. The population forthis assessment was identified as “rising juniors,** that is, students who 
had earned 45 to 59 academic credits. The initial sample of students was drawn at the system level, and individual campuses were 
directed to administertheCAAPtests in writingand mathematical skills to these students. In subsequent administrations ofACTCAAP, 
individual campuses were allowed to select the random sample on a local level. 
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By benchmarkingtest results, UW-Platteville has been able to study both local and national trends in writing and mathematical skills. 
Methodological differences between 1992-the year in which UW System selected the random sample— and subsequent years 
preclude direct comparisons with 1992 results. However, since the mid-1990s UW-Platteville has been able to replicate its 
methodology to the extent that results can be compared effectively. 

Forthe 2001 administration of ACT CAAP, selection of an appropriate sample size was based on the standard deviation of 1998 results.' 
A similar projection was made in 1998, based on the 1996 results, and the 1996 projection was based on the 1994 results. The 
assessment team was able to determine the size of the initial sample required to yield sufficient response by using the following 
formula (with a 95 percent confidence level). 



where n is the minimum sample size needed 

t is the number of standard deviations in the sampling distribution (2) implied by the selected confidence level of 
95 percent 

s is the estimated standard deviation of the variation in the population 
e is the acceptable error 

The response rate of students selected forthe random sample was also projected, based on previous years’ efforts. Anticipating an 
approximate response rate of 50 percent, the assessment team drew a random sample that was twice as large as the minimum size 
required to yield meaningful results. Forthe 2001 administration of ACT CAAP, it was calculated that minimum respective samples of 
45 and 33 were required for the writing skills test and mathematical skills test. UW-Platteville also administered its home-grown 
mathematics exam at the same time, so participants were randomly directed to one of three testing rooms on each testing day. 

A random sample of 271 “rising juniors” was selected from among 507 students in the population of second-semester sophomores. 
Names and addresses of the selected students were provided by the office of the registrar, and each student received a personal letter 
from the provost that stressed the importance of the testing and requested the recipient’s participation. Students were directed to 
either call the provost’s secretary or return an enclosed response card to indicate which of four tests dates was preferred. Based on 
these calls and response cards, a follow-up letter was sent to confirm the time, date, and location of testing. 

Previous attempts to increase response rates among random samples had determined that incentives (e.g., pizza, preferential 
registration treatment) were ineffective, but that personal contact from campus faculty and administrators did yield higher response 
rates. Therefore, the sample of 271 students was divided into ten lists of 27 names, and each member of the assessment team was 
responsible for contacting his or her students by phone, e-mail, or both. Team members were updated about which of their students 
had responded to the initial contact from the provost. The initial letters were mailed approximatelythreeweeks before the tests dates 
(either 4 p.m. or 6 p.m. administration times on different days of the week). Follow-up contacts by the assessment team took about two 
weeks. As a result, 109 students participated in the ACT CAAP and local mathematics test (a response rate of 40.2 percent). Test 
administration took less than an hour each day, with individual examination rooms supervised by a proctor and one floating 
coordinator. 

The completed CAAP writing and mathematics exams were subsequently sent to ACT for tabulation, and results were returned to the 
campus several weeks later. Letters were sent to participants, thanking them for their participation and providing them with their 
individual scores, UW-Platteville averages, and the national norms. 



Data Analysis 

ACT CAAP results were analyzed with a regression model that identified statistically significant relationships between a variety of 
independent variables and students’ test data (dependent variable). In its original design, the regression model employed forty 
variables in its analysis. Variables that proved to be statistically insignificant were eventually deleted from the model. Some of the 
variables determined to be statistically significant in impacting test results were analyzed in more detail; however, the assessment 
team noted that such statistical significance did nof necessarily imply a couso/ relationship. Total variation in a student’s performance 
was the sum of the individual variables’ impact, plus “chance” (for inferential statistics resulting from a random sample). 

Three groups of variables were selected for Inclusion in the regression model. The first group attempted to control for students’ skill 
levels upon entering UW-Platteville. Included In this group were variables such as each student’s score on the entering ACT writing 
and mathematics tests, the score on thestatewide writingand mathematics tests, and whetherthestudent “tested out” of oneor more 
composition classes. 



n = 
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The second group of variables attempted to control for a student's ability, motivation, and other personal and environmental factors 
thought to influence learning. Included in this group were variables such as cumulative grade point average, total credits earned, 
major college (one of three on campus), gender, and whether the student transferred to UW-Platteville. 

The third group of variables attempted to control for policy variables, that is, factors related to the educational experience at UW- 
Platteville. Included in this group was consideration of whether the student had completed coursework in various disciplines (e.g., 
English, mathematics, philosophy, psychology, history). 

Based on previous analyses of ACT CAAP data, several variables were eliminated from the regression model because they were found 
to have no statistical significance. Among these variables were number of English credits completed in high school, high school size, 
and percentile rank in high school. 

Initial administrations of the regression model required the assessment team to input many of the variables by hand. Subsequent 
administrations of the regression model employed the repurposing of data that existed within campus databases (e.g., PeopleSoft® 
and DataMart®). The assessment team employed MiniTab® software in running the regression model. While the “rising juniors” 
sample was not used in administering the ACT CAAP tests for critical thinking skills, the same regression model was employed in 
analyzing students' abilities to think critically. 

Separate reports were prepared for writing skills, mathematical skills, and critical thinking skills each year. These reports analyzed 
the most recent administration of ACT CAAP, compared results to those of previous years, and identified trends in both local and 
national data. Summary reports were provided to the assessment oversight committee, undergraduate curriculum commission, 
academic planning council, and administration (provost, deans, department chairs, et al.). 



Findings 

Without a lengthy explanation of the statistical procedures involved, suffice it to say that variables analyzed were statistically 
significant, and some variables were not statistically significant. Of those variables that were significant, some were significant at the 
.05 level, some at the .01 level, and some at the .ool level. Some variables were significant for one test, but not for another. Most of 
the statistically significant variables followed a pattern from one year's administration of ACT CAAP to another year's administration 
of ACT CAAP. 

The following patterns emerged as a result of regression analyses of the CAAP data over the last few years. Some of these may include 
multicolinearity (e.g., entering ACT English scores and results on the statewide English placement test), which complicated analysis 
of results. 

o On the writing skills exam, students in the College of Liberal Arts and Education scored higher than students in either the 
College of Engineering, Mathematics and Science or the College of Business, Industry, Life Science and Agriculture. 

o On the mathematics skills exam, students in the College of Engineering, Mathematics and Science scored higher than 
students in either of the other two colleges. 

o Less significant in predicting performance on the writing skills exam were cumulative grade point average and entering ACT 
English score. 

o There was a negative correlation (although not statistically significant) between having experienced the second freshman 
composition course and performance on the CAAP writing skills exam. In part, this phenomenon might be explained by the 
fact that 12 percent of the participants had “tested out” ofthe second composition course, possibly suggestingthat they were 
significantly better writers than students who had enrolled in the second freshman composition class. 

o On the critical thinking exam (administered to a different sample, but with results analyzed with the same regression model), 
theonlystatlstlcallysignificantvarlable— and then only atthe. 05 level-waswhetherthestudent had enrolled in a philosophy 
course. Initial interpretation might have suggested that students become better critical thinkers as a result of taking 
philosophy courses. But another plausible explanation was that students who were already good critical thinkers sought out 
philosophy classes because they enjoyed the interactions in such a course. This was a good example of how correlation does 
not prove causality. 

o Freshman performance on the statewide mathematics placement test was statistically significant in predicting performance 
on the CAAP math test for “rising juniors.” Total mathematics credits earned were also significant (.05 level) in impacting 
performance. As a net result, students from the College of Engineering, Mathematics and Science scored significantly higher 
than did students from the other two colleges on campus. 
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o Less Significant in predicting CAAP mathematics performance were variables of gender and cumulative grade point average. 
Given the skewed gender distribution (significantly more males) in the College of Engineering, Mathematics and Science, this 
pattern also seems reasonable in explaining CAAP results. 

o Given the “lead program” role played by UW-Platteville's engineering program, it was not surprisingto find that UW-P*s mean 

on the CAAP mathematics test was usually much higher than the national norm. 

o Asignificant increase in CAAPwritingscores between 1994 and 1996 came on the heels of a shift ofthe pedagogical approach 
in the freshman composition courses. 

While not exhaustive, these results suggest that regression analysis of ACT CAAP data can yield valuable insights into students* 
performance on standardized tests. As input for curricular and other administration decision making, such analysis offers new 
perspectives and confirmation of previous assumptions regarding our students. 



While standardized tests are not a panacea for all institutional problems, the analysis of data can provide new ways of looking at 
students* abilities and performance. Software programs designed to minimize calculations formerly conducted by hand make 
regression analysis easier to perform today. And the growing use of databases of student information makes it much easier to access 
a wide range of variables, often through the repurposing of data. 

Regression analysis can be a cornerstone in institutional analysis of standardized test data. Results of such analysis can offer new 
opportunities for institutional research, as well as provide greater insight into predicting student performance at the college level. 



*For example, see Morris Hamburg, Statistical Analysis for Decision Making (San Diego: Harcourt, Brace, Jovanovich, Publishers, 
1991), pp. 300-302. 



John C. Simonson is Professor of Economics at the University of Wisconsin-Platteville. 

George E. Smith is Professor of Communication Technologies at the University of Wisconsin-Platteville. 
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A Handbook of Assessment 
for the Layperson 

Kenneth R. Ryalls 



Initial Assessment Efforts 

Forthe past five years I have been chair of the assessment committee at mycollege.andinthat role! have attended many conferences 
on assessment. During those conferences I have learned much about the rules and practice of assessment on a theoretical level, but 
1 found that my assessment education rarely addressed the more practical question of implementation. Assessment discussions 
almost always left me with questions: “Yes, but how6o 1 realistically implement these valuable and worthwhile ideas? What are the 
difficulties faced by the person responsible for the coordination of assessment? How do 1 educate (and motivate) the program 
directors, division chairs, and area supervisors to be willing and able to actually do the assessment?” Unfortunately, I never got 
answers before embarking on the challenge of implementing meaningful and ongoing college-wide assessment practices, and I had 
to learn through trial-and-error about the realistic challenges of such a task. 

Oneofthe biggest challenges that institutions face when workingon assessment is to get faculty buy-in to the process, because faculty 
often see assessment as an unnecessary (and potentially punitive) crosscheck of their work. To increase ownership of the assessment 
process amongst faculty, many colleges (mine included) decide to put assessment directly into the hands of the faculty, letting each 
program of study design and implement its own assessment practices. In this way, program directors (with the assistance of their 
faculty) determine what is important in their own programs and then design a system of how to assess those goals on a program-by- 
program basis, rather than being dictated to by a centralized governing body of assessment. (Note: For ease of communication, 1 will 
refer to those responsible for assessment as “directors,” rather than creating an exhaustive list of possibilities, such as program 
chairs, division chairs, managers, deans, or VPs. 1 also maintain the focus on assessment of academic programs for simplicity, 
although the principles discussed here extend beyond the academic area.) Decentralization of the assessment process is a great way 
to achieve greater faculty commitment to assessment, but this strategy has an inherent problem associated with it: the more 
decentralized the assessment process gets, the less controlthe assessment “czar” atyour institution has overthe assessment efforts. 
Perhaps even more difficult (and costly to the institution) is that an increase in people directly responsible for assessment means an 
increasein people who need to be fullytrained in the art of assessment. Couplethat problem with natural turnover amongst directors, 
and a major problem arises. Even with the best of intentions (which is often not the case), assessment efforts languish without stable 
and consistent guidance across time. 

So the obvious response is to centralize assessment efforts and make one person or office responsible for the assessment of all 
academic programs on campus. Butthen you have people responsible for assessing learning outcomes in programs thatthey arguably 
kn ownothingabout, and a faculty that is inherently mistrustfulof an outsidercomingin and measuring performance. Ifyourinstitution 
is anything like mine, you cannot (and probably should not) pay a full-time person to coordinate all assessment efforts on campus. 
So what*s the answer? This is a question I have struggled with during my first four years as the coordinator of assessment on campus, 
implementing many potential solutions and learning firsthand what works and what does not. I cannot give you the answer, but I can 
give you an answer— and so far, the answer that we adopted seems to be working well. The answer is based on a few fundamental 
assessment truths: 

o The assessment committee needs real authority. 

o Those responsible for assessment need readily available assessment education. 

o Basic assessment procedures need to be standardized across programs. 



The Need for Authority 

The first truth does not relate directly to the Handbook. However, I believe that it is a truth worth mentioning because it was the most 
painful to learn, and it is necessary to know if the Handbook is to have any real impact. My initial approach to assessment was guided 
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by a naive belief that directors would be more than willing to put in the necessary work if 1 could convince them that assessment was 
a worthwhile endeavor. As any competent manager knows, there will never be great love for a system whose purpose is to show 
employees where they are coming up short, even if the employees truly believe it is a good idea. It is also difficult from a managerial 
perspective to motivate people to put in extra time and effort out of the goodness of their hearts. An additional problem at our college 
was that our assessment committee was a faculty committee and therefore did not have any real authority over how assessment was 
done. Rather, our role as a committee was to suggest improvements on a yearly basis and try to affect change through that yearly 
feedback. We would then hope that the changes were indeed being made over the course of the year, a process my boss described 
as “touching” when he first came to our college. Needless to say, suggested changes from the assessment committee were often 
ignored; similar plans were handed in for review the next year; and similar feedback memos would again be sent to the program 
directors, requesting the same changes as the previous year. Change was glacial at best. 

To combat this problem, an assessment committee needs teeth in the form of true authority. The approach our vice president of 
academic affairs (VPAA) adopted was to make the assessment committee his own, rather than a faculty committee. Because we report 
directly to him, memos sent by the committee are now bolstered by the authority ofthe VPAA, which makes all the difference in affecting 
change. No longer does the assessment committee need to wait a year to see if suggestions have been followed. Instead, our memos 
address specific changes that need to be made, along with a date by which the changes must be made. The assessment committee 
can now affect fairly rapid change, bringing even the weakest plans up to a basic level of competence within a year. 



Educating the Masses: The Birth of the Handbook 

Once effective assessment governance is in place, the burden of actually doing the assessment falls upon the directors. Ifyour college 
is similar to mine, you will find that most of your directors have good hearts and the best of intentions about seeing the assessment 
process through. What is lacking is the knowledge to do it. At the initial stages of design, assessment is a difficult and complicated 
process, especially for a person with little formal training. Many colleges have thirty or more program directors, and the taskof quickly 
and easily bringingthem all up to speed becomes critical to the success ofthe assessment effort. Our institution initially tried to train 
the assessors with in-service meetings offered once or twice a year, but that approach has some problems associated with it. To begin 
with, in-service is often a lecture, and people do not come out ofthe meetings with perfect retention of information. You can try to aid 
memory by providing handouts to supportyour lecture, but those handouts are often incompleteand end up being lost orthrown away. 
Common handouts include the standard “Assessment Cycle,” a bulleted list ofthe required components of assessment according to 
the Higher Learning Commission, and perhaps contact information for the members ofthe assessment committee. Great information, 
to be sure, but certainly not enough to let your directors hit the ground running. In addition, in-services are usually offered only once 
a year, yet personnel and responsibilities are constantly in flux. I continually ran into the problem of how to train the new directors 
without losing a year of assessment in the process. Although face-to-face meetings can solve the problem, my main function on my 
campus is not coordination of assessment, and time constraints limit how many personalized training sessions 1 can provide. After 
a few years of trying to keep assessment efforts going. Truth Two became increasingly evident: There has to be a comprehensive 
ongoing system of assessment education in place. What 1 needed was an assessment primer to distribute to anyone who needed to 
knowthe basics of assessment, and needed to knowthem immediately. 1 was bettingthat the time it would take to write such a primer 
would be offset by the time savings it would provide in the future. Thus, the Handbook ms born. 



Elements ofthe Handbook 

Through experience, I knew what parts of assessment directors found particularly difficult, so it was fairly easy to generate a table of 
contents for the Handbook. Any useful handbook would need 

0 An overview ofthe assessment process. Before getting started, all assessors need to know what is expected of them. How 
does the process of assessment work? Why are we doing it? When are reports due? Who reviews them? This section is specific 
in expectations and has a clearly defined timetable of assessment, along with identification ofthe specific reports that are 
expected during the course of each year (summary, assessment report, and year-end report). 

0 The basic requirements of a good assessment plan. Although directors should have leeway in designing their assessment 
plans, there are basic requirements for every plan. For example, program goals should flow from the program’s purposes, 
which should in turn flow from the institution’s mission. But unless someone is very knowledgeable about assessment, how 
could we possibly expect the average director to have an understanding of that process? Rules and requirements that are 
second nature to those ofus immersed in assessment arecompletely foreign to the average person and need to be specifically 
identified. 

0 General advice. Every handbook should have a section like this, because it provides a place to record and pass on those often 
unwritten rules of assessment. Experience breeds wisdom, yet we rarelytakethetimeto record lessons learned so that others 
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may benefit from the painful lessons that weVe learned. Yet experiential knowledge is often more valuable than the official 
information found in guidelines and instructions, and it should be communicated. For example, one of the common sins 
committed when beginning the assessment process is to make the plan unnecessarily complicated. It is a natural mistake, 
because once directors start to list the goals of their program, they tend to create a list of all of the hopes and dreams they 
have for their students. 1 have seen preliminary plans that have fifteen or more goals for students, which necessarily leads 
to a plan of assessment for all of those lofty aspirations. Obviously, one director is probably not going to be able to actively 
manage the assessment of fifteen different goals. Therefore, the first bit of advice given in the Handbook is to keep the plan 
as simple as possible. 1 also use this section to address common complaints associated with assessment, such as the 
complalntthatassessmentis unnecessary busywork. When directors reallzethat a program is much more llkelytogetfundlng 
for a needed item if the request is supported by assessment data, the value of assessment suddenly becomes clear. 

0 Construction of measurable goals. Each program should have a mission and/or purposes statement, and the goals of the 
program should flow out of that statement. The creation of goals sounds fairly simple, but it is often difficult for directors 
because first attempts at mission statements tend to become unnecessarily comprehensive and cumbersome. In addition, 
first attempts at construction of program goals are almost never objective and measurable, leading to frustration when the 
director tries to design the proper tool for assessment. When they are provided with a step-by-step guide on the writing of 
mission, purposes, and goals, along with specific examples illustrating each step, directors have a much less painful time 
designing their own goals. 

0 Catalog of available tools of assessment. One of the most common questions 1 get, time and time again, is how to assess 
the more subjective goals. Much of what we do as educators is abstract-ideas such as critical thinking, effective 
communication, and college-level skills are very difficult to measure. But difficult does not mean impossible, which is the 
claim of many directors. When the director of the art program says that one of his goals is to make sure that all students have 
an aesthetic appreciation of modern art, then says that there is no way to assess aesthetic appreciation, the entire attempt 
atassessmentgrindstoa halt. Agood handbook should identify the ways thatwe could possibly measure appreciation (such 
as an expert's review of the student's final paper on modern art), and help him to see that it is Indeed possible to do so. The 
tool catalog also points out the pros and cons of each available assessment tool so that the assessor can make sound 
decisions about which tool to employ. 

0 Effective use of data. Because I am by training a social scientist I have a fairly strong understanding of the use of data, 
especially data that contain a lot of variability and fluctuation over time. However, I find that many directors tend not to have 
the background necessary to make data-based decisions in a cautious and rational way. In fact, mostof my faculty colleagues 
lie in one of two camps: those who believe data are useless, and those who believe that data never lie. So the reaction to 
negative data collected is either to ignore the information because it is wrong, or to change our entire approach to education 
because the ship is sinking. The truth is that data can be useful only if interpreted correctly, and the goal in this section of the 
Handbook is to educate directors on the wise use of data. This section addresses the options available to the director, so that 
we are neither frozen by inertia nor prone to knee-jerk reactions to every random fluctuation of data that we record. 

0 Brevity. A useful handbook would also need to be brief. Assessment is rarely anyone's full-time job, so no one wants to read 
a hundred-page handbookon proper assessmenttechniques. Mygoal was to create a comprehensive handbookthatwas brief 
and easy-to-read, and communicated all that I knew about basic assessment practices. Currently, the entire Handbook is 
thirty-three pages long, including examples and templates. The meat of the Handbook, though, is only fifteen pages long— 
a digestible size for the average reader. 

0 An assessment report template. Truth Three (the need for standardization of assessment practices) arose out of necessity. 
There was no way 1 could provide a brief and painless primer on assessment if there wasn't some standard that all directors 
were required to follow. With a standard assessment reporting style required of all programs, 1 could be much more specific 
in describing how assessment is done. An added bonus of a standardized assessment reporting style is that it is much easier 
for the assessment committee to review, as all reports look Identical in content and form. 

0 A complete example of a quality assessment report. Finally, there is nothing more helpful to a first-time assessor than a 
complete working example of what is expected. Contained within the Handbook are the assessment summary, assessment 
report, and year-end report of a fictitious interior design program. A fictitious program was used deliberately, so that all 
directors could see the report through eyes free of expectations. If an existing program were chosen as an example, there was 
potential for the directors in disciplines similar to that program to feel obligated to make their assessment report Identical 
to the example. 



Initial Results 




The Handbookms adopted at the start of the 2002-2003 academic year, and so far the initial results have been quite encouraging. 
Those directors who have taken the time to read the Handbook have made remarkable progress in the construction of a viable 
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assessment plan. Many ofthe directors who had submitted weak assessment reports in the past have already submitted clear, concise 
assessment plans for this year. Feedback on the Handbook from directors has also been quite positive, and one actually suggested 
that I submit the Handbook for presentation or publication (a suggestion I took to heart). However, the Handbook is not a panacea for 
all problems we have had in the past. There is still resistance to the assessment effort from some ofthe directors, resulting in late or 
incomplete submission of plans. But the Handbook does provide a common point of reference when communicating with the 
stragglers, because I can now provide instant feedback and refer the director to the specific page that fully explains the process to 
follow. With the authority of the vice president's office supporting the assessment committee, we can get turnaround and a plan's 
resubmission within weeks ofthe request. Finally, like assessment itself, the Handbook is a continually changing document, and will 
shift in scope and content as dictated by the experiences of those using it. Hopefully, it will continue to have a positive impact on the 
assessment process at my institution for many years to come. 



Kenneth R. Ryalls is Chair of the Division of Arts and Sciences and Director of Psychology at the College of Saint Mary in Omaha, 
Nebraska. 
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Activating Assessment of Student Learning: 
A Systematic and Systemic Approach 

judith Neill and jeanne Williams 



Context 

Moving from plans for assessing student academic achievement to action and results presents many challenges for colleges 
committed to a process improvement philosophy. Toolsthat support the entire cycle ofplanning, delivery, assessment, reflection, and 
continuous improvement can play a key role in effective deployment and documentation. 

Blackhawk Technical College (BTC) in Janesville, Wisconsin, completed its on-site review for reaffirmation of accreditation in March 
2000. As BTC faculty and staffworked to implementthereviewteam’s recommendations, they recognized the need to think differently 
about how they could use technology to support and document the process. The Higher Learning Commission requested a progress 
report by July 2002, giving BTC a window of opportunity to develop the reflection and continuous improvement phases of the 
assessment cycle. The Commission approved the progress BTC reported in its July 2002 follow-up report. 

Critical to the completion of BTCs student outcome assessment plan was interpretation and use of the “Levels of Implementation” 
proposed by Commission staff liaison Cecilia Lopez, as wellas a matrixtool developed and used to determine when learningoutcomes 
are assessed in the program cycle. Staff at BTC provided instructors with opportunities for professional development and clear 
communication about the purpose of these tools for enhancing the institution's plan and follow-through activities to move forward 
and put the plan into place. 

Beginning in 1993, the sixteen colleges that make up the Wisconsin Technical College System (WTCS) collaborated to develop the 
Worldwide Instructional Design System (WIDS), an outcomes-driven learning design model, software, and professional development 
program based on research and best practice. To that end, frontline teachers and learning designers continue to design, implement, 
and continually improve WIDS. (From 1993 to 2001, WIDS was known as the Wisconsin Instructional Design System.) 

BTC has been an active partner in the development and continuous improvement of WIDS since its inception. At the time of its on-site 
reviewin March 2000, the college had made significant progress in usingthe WIDS model and software to develop college-wide core 
abilities, program outcomes and measures, and course-level competencies supported by performance standards and rubrics or 
checklists. 

As staff began to accumulate and analyze data for the report, BTC simultaneously explored systems that would support instructors 
and administrators in reporting assessment data and results, and assist the institution in getting its arms around the large volume 
of information that is generated in the assessment process. At this time BTC administration made a decision to expand their toolbox 
by adding TracDat Assessment Management software, a Web-based software package that can be accessed by administrators and 
instructors without regard to location and time. The application provides a central location and tool for documenting and reflecting 
on implications for continuous improvement, and actions taken in response to the reflection. 

BTC uses SCT Banner as their enterprise-level software system for managing courses, registration, and student services. By 
envisioning a way to put WIDS, TracDat, and SCT Banner together, BTC is attempting to implement a seamless approach to an 
integrated assessment process. 



The Underlying Model 

Recognizing the need to establish a framework for thinking, communicating about, implementing, and reflecting on learning and 
assessment, the Wisconsin Technical College System crafted a model depicting the constructs and the process. While acknowledging 
that an effective framework may take many forms, they concluded that a clearly defined, consistent framework would be helpful in 
moving forward and sustaining the process. The resulting model Is based on proven research, examples of excellent practice, and 
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input from frontline educational leaders. Though it is unique in detail and terminology, it represents a synthesis of principles and 
constructs of instructional systems design (ISD); competency or performance-based learning; and learning-centered teaching, 
learning, and assessment. 

To grasp a quick overview of the model, think in terms of four concentric circles. In the center is “WHO,” representing the question, 
“Who are mylearners, and what are their learning needs?” The next circle is “WHAT,” representing the question, “What do my learners 
need to be able to do as a result of this learning experience (e.g., course or lesson), and how does what they learn here need to relate 
to a broader context (e.g., program, major, or degree)?” The third circle is “WHEN,” representing the question, “How will my learners 
and I know when they have learned what they needed to learn?” The outermost circle is “HOW,” representing the question, “How can 
the learners and I best meet the goals set by WHAT?” 

As with most working models, there are more complex layers and unique terms to describe them. Those more salient to understanding 
the assessment process in terms of the WIDS model differentiate between varying levels or layers of learning outcomes; 

0 Core abilities are macro outcomes made up of transferable skills, attitudes, and abilities to be mastered by learners 
completing a sustained course of study such as a program, major, or degree. Though they will be partially learned and 
assessed at the course or lesson level, core abilities go beyond a specific course. They are integrated throughout the learning 
experience, and are often determined at the organizational level. A group of core abilities is derived from the organization's 
mission and purpose. 

0 General education outcomes are macro outcomes made up of general education (academic foundation) skills, knowledge, 
and attitudes to be mastered by learners completing a program or degree. Like core abilities, general education outcomes are 
partially learned and assessed at the course or lesson level, but usually go beyond a specific course. A group of general 
education outcomes generally pertains to a degree level such as bachelor's degree, associate degree, technical diploma, or 
certificate, or to a program. 

0 Program outcomes are macro outcomes made up of /?e/c/-spec///c skills, attitudes, and abilities to be mastered by learners 
completing a program or major course of study. Like core abilities and general education outcomes, program outcomes will 
be partially learned and assessed at the course or lesson level and go beyond a specific course. A group of program outcomes 
pertains to an entire program or major. 

0 Competencies are major skills, knowledge, or attitudes that are measurable and observable. When combined with 
performance standards (criteria and conditions), they equate to Higher Learning Commission “measurable objectives.” 
Competencies may be academic or field-specific. They are fully learned and assessed at the lesson or course level. To show 
that a specific competency addresses a macro learning outcome (or inversely, to show that a macro learning outcome is 
embedded in a competency), designers link the macro learning outcome to the competency with the intent that it will be 
learned and assessed in conjunction with the competency. 

Assessment is the process of establishing learning outcomes that meetthe learners' needs (what), determiningwhetherthey have met 
them (when), documentingthose results, and reflectingon howto continually improve the process ofteaching, learning, and assessing. 
The purpose of the assessment process is to continually improve and document or credential learning. The WIDS model enhances the 
assessment process by providing a framework that supports thoughtful planning; communication to all stakeholders of defined 
measurable outcomes at the beginning of the teaching and learning process; deployment of valid, reliable, and fair assessment 
strategies; informed reflection on the results; and continuous improvement ofteaching, learning, and assessment. A key feature of 
the WIDS model is the links that are established between outcomes learned at the lesson and course levels with broader learning 
outcomes representing complex, integrated skills and abilities learned during the course of a sustained learning experience (e.g., 
program, major, or degree). 



Software Toolbox 

The WIDS software provides the front-end instructional design tools, while TracDat offers tools for recording, organizing, analyzing, 
and reporting assessment data that support reflection and continual improvement. BTC employs SCT Banner enterprise system as the 
backbone data management structure for the college. SCT has recently partnered with Nuventive-the company that developed the 
TracDat software. This new partnership presents an opportunity for BTC to integrate data from the administrative data structure into 
the learning and assessment systems that the institution is putting into place. 
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The vision is to be able to bring components designed using the WIDS model and software, and data currently residing in the 
administrative enterprise system, into TracDat in order to manage the complete plan,-do-check-act cycle of assessment that results 
in continuous improvement ofteaching and learning. At this point, BTC is engaged in the early stages of deployment, and staff 
members are transferring data among the three software packages manually. 
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o TheWIDS learningDesign software featuresthree distinct modules (Program Design, Learning(course) Design, and DACUM) 
that can be used independently or in conjunction with one another. Designed to assist teachers and instructional designers 
with the front-end planning of learning and assessment, it provides a consistent model; design templates; and libraries of 
outcomes, learning strategies, and assessments, along with context-sensitive coaching. Output, published in Microsoft 
Word, HTML, or rich text format Includes learning plans (modules), teaching plans, complete syllabi, outcome summaries, 
program designs, DACUM charts, assessment/outcomes matrices, and many others. WIDS tools support important parts of 
the plan-do-check-act elements of the cycle, while the teacher further draws this circle by deploying these elements on the 
frontline learning environment and reflecting on the results. 

o TracDat is a software solution for managing the academic assessment process. Designed to support all phases of the 
assessment process, TracDat facilitates the use of assessment data for planning and improvement. By supporting develop- 
ment of assessment plans and the identification of program strengths and weaknesses and recording corrective actions and 
follow-up results, it documents the impact of assessment on academic programs. TracDat plays an important role in 
completing the plan-do-check-act cycle by integrating intended learning outcomes designed in WIDS and student data that 
currently exist within the enterprise system with assessment plans and results documented in TracDat. 

o SCT Banner is an enterprise-level software suite that provides applications for student information, advancement, human 
resources, financial aid, faculty and advisers, and finance. 

Reflections 

Colleges that strive to live by learning-centered principles must address many challenges and issues. The ability to move from plans 
for assessing student academic achievement to action and results is critical in reaching key milestones. In our work with frontline 
teachers and instructional designers in the two-year college and university environments, three barriers to moving forward with the 
activation of assessment of student academic achievement stand out. 

The first barrier is the tendency for each organization, group, and even individual to recreate the model, using unique constructs and 
terminology. In our work with educators throughout the nation, we’ve observed that teachers and instructional designers frequently 
invest a great deal of time and a significant proportion of their instructional design dollars spelling out new frameworks and 
terminology-^literally “rediscovering the wheel.” Arguably, the process of defining a model provides a critical thinking learning 
experience and local acceptance of the model. However, the result has been a bit like the Tower of Babel, leaving the educational 
communitywith the perception that we cannot workwith a common frameworkand languageforinstructional design and assessment. 

When we began building the WIDS model, we believed that we would have a very difficult time aligning the instructional design 
frameworks of sixteen colleges. We found instead that the primary differences lay in our language for describing the elements of 
instructional design and the level of detail to which teachers and designers take the components. As we benchmarked “our” model 
with other learning organizations across the country and around the world, this findingwas reinforced. We believe that educators who 
proceed with a “adopt, adapt, or revise” perspective are better able to focus the use of scarce instructional design resources on the 
design, deployment, assessment,and continuousimprovement of specificlearning experiences(degrees, programs, majors, courses, 
workshops, etc.) rather than continually reinvesting them in a never-ending cycle of reworking the model. 

The second barrier is a disconnection between planning and practice. As most colleges move toward assessment of student learning, 
members at all levelsofthecollege community engagein much activity, settinggoals,publishingdocuments that describe and support 
the initiative, and establishing macro learning outcomes. Unfortunately, the plans sometimes have little impact on what happens on 
thefrontlines ofteachingand learning. Colleges that integrate the assessment process with the instructional design process are more 
successful in making assessment of student learning a way of life In the educational community; and making assessment of student 
learning a way of life is the only way to ensure that it truly can make a positive difference in the quality ofteachingand learning. 

The third barrier occurs when educators take the position that they need to build their own software tools in order to achieve their 
unique goals. It is easy to underestimate the investment needed to initiate and nurture a “home-grown” software tool. Each of the 
software tools presented In this case study cost millions of dollars and took several years to develop. They are the products of much 
research and input from thousands of practitioners. Educators who are willing to examine existing software tools and choose one that 
they can adopt or adapt to meet their needs will avoid the quagmire of developing tools that require the collaboration and the 
combined resources of many organizations. Though no one tool is right for all organizations, chances are good that most colleges will 
be able to find an existing tool that can help them do the job. 



Judith Neill is Director of Instructional Design and Planning at the Wisconsin Technical College System Foundation in Ripon. 

Jeanne Williams is Curriculum/Alternative Delivery Specialist at Blackhawk Technical College in Janesville, Wisconsin. (Prior to this, 
she worked with the Distance Education (IDE) project at Penn State University. 
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Educating Seniors, Assessing Ourselves: 
A Senior Year Experience 

Beth Rigel Daugherty, Dan Thompson, and Susan Thompson 



Introduction / 

The Otterbein College Senate instituted the Senior Year Experience (SYE) requirement as a pilot program in January 1995, charged the 
ad hoc committee responsible for creating the requirement with implementing and testing it for four years (classes of 1999-2002), 
and called upon the curriculum committee and the senate to decide at the end of the experiment whether the SYE requirement should 
be “continued, transformed, orconcluded.” At that moment, a committee that had been focused on creatinga curricular space for the 
integration of the major and the general education core became an assessment committee as well. This presentation will explain how 
a group of people with no formal training in assessment developed assessment tools, met the need for qualitative and quantitative 
data, and created an assessment model for continued improvement. More important, our presentation will show how the collecting 
of assessment data transformed us: how the frightening demand that we prove our worth at the end of four years became, instead, 
a sustaining process that taught us a great deal about our seniors, our program, and ourselves. 



The Senior Year Experience Requirement at Otterbein College 

otterbein is a private, church-related, four-year coeducational college located in Westerville, Oh io, a suburb of Columbus. The college 
sponsors traditional and continuing education programs in liberal arts and professional education at the baccalaureate and master’s 
levels. Its mission is to “educate the whole person in a context that fosters thedevelopment of humane values.” Fifty-one percent of 
its students choose professional majors, putting Otterbein in the “comprehensive” category used by Carnegie. All students must take 
ten courses in the integrative studies program, a liberal arts core focused on the theme of human nature. The IS program spans the 
four years of the undergraduate degree and requires students to take courses in the social sciences, the arts, the natural sciences, 
religion and philosophy, including global thought, and literature and composition. 



TheSeniorYear Experience requirement grew out ofsurveysofseniorsand alumni conducted in the lateigSos that suggested the need 
for a culminating curricular segment at Otterbein. Many seniors stated they did not connect the major and the core curriculum, and 
very few felt they had an improved understanding of contemporary social issues. Alumni indicated they lacked practice in doing 
interdisciplinary work and requested opportunities for gaining such practice. After extensive experimentation with interdisciplinary, 
team-taught courses in 1992-1994, positive student and faculty response to those courses, and campus-wide discussions, theSenior 
Year Experience requirement was developed and then approved as a pilot program by the college senate in 1995. SYE was created to 
' provide a placein the curriculum where students and faculty could striveto reach the broaderinstitutional goals stated in the college’s 
mission statement: its commitment to “the liberal arts as the broad base of all learning,” its support of lifelong learning in which 
students “develop the ability to participate thoughtfully in discussion and decision making” and “develop the powers of synthesis,” 
and its aim to create “an atmosphere which stimulates students to become aware of themselves and their responsibilities within a 
larger, multi-cultural society” and prepare “the whole person to develop responsible commitments to society.” 



The Senior Year Experience is designed to teach and strengthen the skills and practice of synthesis; to ensure structure, content, and 
credit for the work of connecting learning across disciplinary boundaries; to provide a focal point and closure for the four years of a 
liberal arts education; and to provide a bridge between academic learning and responsible citizenship. SYE courses and options were 
developed to expose students to significant contemporary social issues that challenge us as citizens; give tools for understandingand 
making ethical choices when faced with such issues; provide a curricular space in which to bring togetherthelearninggained in majors 
and integrative studies courses; provide opportunities for working together on issues that require a wide range of disciplinary 
perspectives and expertise; and enable students to consider their roles as educated persons, professionals, and citizens. The SYE 
requirement also provides a way for the college to assess itself. Students can meet the SYE requirement through (1) team-taught five- 
credit-hour interdisciplinary SYE courses; (2) departmental options that meet the SYE criteria; and (3) SYE options tied to off-campus, 
immersion, internship, or individualized experiences. 
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Program goals for the SYE requirement had been developed at the outset and were stated in the original proposal to the campus 
community. The SYE committee charged with implementing, delivering, and assessing the SYE requirement has expected all SYE 
offerings to 

o Analyze complex contemporary issues, for which the liberal arts graduate is expected, by society, to provide leadership 

o Develop, refine, and practice the skills of synthesis, ensuringthat students drawfrom multiple disciplinary bases and move 
beyond relativist assessments of diverse perspectives to responsible synthesizing of these perspectives 

o Use /r?fer(//sc/p//r;arycoursestructuresand materials, emphasizingthattheliberalartsgraduateneedstobeableto recognize 
the strengths and the limits of different disciplines 

o Structure active self-reflection, including conscious attention to ethics and personal values 

o Provide models and structures for students to integrate learning, bringing to bear knowledge gained in core courses and 
knowledge derived from major and elective courses 

o Bring students into social and intellectual interaction with the wider community beyond Otterbein*s campus 



These program goals were communicated to faculty participating in summer workshops on SYE course development and on 
interdisciplinary and team teaching. All proposals to the SYE committee for courses, departmental options, or immersion options had 
to show how these program goals would be met. 



Multiple Assessment Measures for SYE 

Our development of assessment tools looks much more coherent in retrospect than it did while we were developingit. In practice, the 
process was recursive; we were constantly revising on the basis of new knowledge or previous assessment. The following summary 
does not reflect the numerous hours spent sitting around a table discussing student learningoutcomes;thewordingof questions and 
statements on the tools described below; and our uncertainty about whether this type of student learning could be measured, and 
if so, how. How were we to measure the learning of intellectual skills such as synthesis, for example? Our instincts told us we had a 
good idea, that we had created a good addition to our curriculum, that we were basically doing a good job in implementing and 
delivering it, and that students were learning what we wanted them to learn. But we couldn't take those instincts to the bank. 

Earlier surveys had showed us that alumni generally did not see the value of integrative studies, the liberal arts core at the heart of 
Otterbein's education, until five to ten years after graduation. In fact, that realization had had a lotto do with the creation of SYE in 
the first place. Would SYE help alumni see the value of their liberal arts education sooner? That question underlay much of our work, 
but two other questions guided the creation of our assessment tools: (i) Does SYE make a difference? (2) Do students who fulfill the 
requirement reach SYE goals? The first question implied two sorts of measurement: (1) baseline data collected prior to SYE that could 
be compared to data collected during the years of the requirement and (2) data that would measure the impact of SYE. The second 
question implied a more specific kind of measurement and meant focusing on student learning. In addition, we had tried to create a 
curricular space for bringing together the major and integrative studies that would be experienced by students as different from both. 
Had we succeeded? For that and many other reasons, we needed to knowwhat students and alumni thought of SYE. Later, we realized 
we needed to know what the faculty who were developing and delivering SYE thought as well. 

Sinceourinstitution was already administeringthe Higher Education Research Institute freshman year survey to its freshmen, theSYE 
committee decided to administer the nationally normed College Student Survey to seniors beginning in 1996. We hoped the CSS, 
combined with the CIRP, would give us reliable baseline data for tracing the attitudinal change from freshmen to seniors both before 
and duringthe years ofthe SYE requirement. Because some committee members had taken part in a FIPSE-sponsored critical thinking 
project in the mid-1980s and had studied William Perry's process for creating his developmental schema, we decided that videotaped 
senior exit interviews, conducted both before and after the requirement began, might also be useful for collecting data from before 
and during SYE. 

We used our college's standard course evaluation form to assess student reaction to SYE offerings, but we also developed a set of 
additional questions more directly related to SYE to assess students' perceptions of their own learning. Although the time frame of 
the pilot program did not give us many alumni to consult, we did an alumni survey of two classes. We collected student comments from 
the course evaluation forms, from reflective journal entries and papers, and from self-assessments conducted in some classes. We 
gathered additional student feedback in student focus groups held right before graduation and faculty feedback in faculty focus 
groups. Finally, at mid-point and near the end ofthe experiment, we sought the help of Otterbein’s assessment committee and three 
cof?Si//for?fs.Weweregatheringagreatdealofdata,andaseducatorswethoughtweknewwhatthedataweretellingus,butweneeded 
help in (1) analyzing and interpreting the data accurately, (2) converting masses of qualitative data into quantitative data and putting 
all data into formats that others could read and understand, and (3) assessing our assessment efforts. 
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Always guiding our assessment efforts were the six program goals for SYE offerings noted above. Everything from extra questions on 
the CSS to the senior exit interview prompt, from extra questions on the course evaluation form to the alumni survey, from the 
questions we asked students and facultyto the questions we asked our consultants grewoutofour program goals. We had many other 
questions, of course, and the questions kept changing, as happens in all human enterprises. But we kept pulling ourselves back to 
our purpose and our goals. Somehow we knew, even as assessment novices, that those goals were our anchor, were what we had to 
measure, and were tied to what we wanted students to learn. 



Using Assessment to Measure Impact 

Does SYE make a difference? D \6 student learning after the requirement began differ from student learning before the requirement? 
The College Student Survey could not tell us. For one thing, because we could administer the CSS on a voluntary basis only, the 
percentage of students completing both it and the CIRP was extremely small. For another, CSS measures student attitudes about their 
college experience and about specific social and political Issues rather than outcomes related to SYE goals. Dr. Karl Schilling, the 
external evaluator we consulted to assess our assessment tools, advised us to discontinue its use. He also suggested that our senior 
exit interviews were valuable, but that our home-grown coding methods were not giving us the data we needed. As a result, we 
consulted Dr. Sandra Pritz of the National Occupational Competency Testing Institute (NOCTI), who developed a rubric based on the 
goals of the SYE program. She and two NOCTI project staff members viewed and coded the tapes from 1997 and 1998 (pre-SYE years) 
and 1999 and 2002 (SYE years). Their analysis showed clear evidence that SYE makes a difference in students* abilities to synthesize, 
integrate the major and the general education core, and reflect on the whole of their education. SYE also increases students* 
recognition of the strengths and limits of different disciplines and increases their engagement beyond the campus. 

Do students reach goals? The interviews, a direct measure of student learning, told us that over 85 percent of our seniors either 
moderately, clearly, or significantly met all the SYE goals during the SYE pilot program. The additional SYE questions on the course 
evaluation form, on which students reported their own perceptions, told us that the SYE offerings succeed in helping the majority of 
them make connections, confront contemporary social issues, and think about themselves as citizens. Results of the alumni survey 
oftheclasses of2ooo and 2001 correspond to the results ofthe externally evaluated senior exit interviews and the self-reported course 
evaluation data for the additional SYE questions: SYE expected students to confront contemporary social Issues, helped them make 
connections, moved them to engage with the community beyond Otterbeln, and helped them begin to think about themselves as 
citizens. A majority of students and alumni reported that SYE had helped them be more open to new ideas, provided the opportunity 
for interaction with a more diverse group of students, provided a different educational experience from other courses taken at 
Otterbeln, and had added something positive to their education. 



Using Assessment to Improve 

The consistently positive response to SYE did not mean, of course, that the requirement was having an Impact on every student orthat 
negative feedback was nonexistent. Course evaluation data, specific item means for the SYE additional questions on course 
evaluations and on the alumni survey, student and faculty focus groups, and comments gathered from a variety of sources gave us 
information about concerns and told us that their nature was also consistent. In fact, the negative feedback reinforced the positive, 
giving us the same messages about what we needed to Improve. Reading the data has meant responding with change; thus, 
assessment has shaped the program*s evolution and its future direction. 

0 Program. In his evaluation of Otterbeln*s senior year. Including SYE, Dr. John Gardner told us that the program*s academic 
goals were solid, but that we needed to pay more attention to the transition seniors were experiencing. Faculty members told 
us that students did not seem to understand the purpose of SYE. As a result, the SYE committee clarified the nature of SYE— 
that it is more of a bridge than a capstone-and worked with the registrar, the career development center, student affairs, and 
development to create a senior survival guide. This guide, which is given to juniors prior to spring registration, includes 
sections on the nature of transitions, deadlines, SYE (with more student-friendly descriptions ofthe program, offerings, and 
choices), careers, and celebrations for seniors campus-wide. In addition, we added a session on the senior transition to the 
summer SYE workshop for faculty. We plan to continue to work with the rest ofthe college to draw attention to the senior year 
as an important transition that requires thoughtful curricular and co-curricular planning from all of us, and our next project 
is a Finances 101 booklet and/or workshop. 

Students noted difficulties with schedulingand expressed dissatisfaction with lackof choice in SYE offerings. They also were 
disappointed when offerings lacked currency or contemporary relevance. As a result, we have stepped up efforts to recruit 
faculty for SYE; have publicized the choices that are available more widely, including in [he senior survival guide; and have 
tried to ensure that SYEs are offered in all our delivery systems throughout the academic year. We have also clarified SYE*s 
focus on citizenship and emphasized the importance of contemporary topics with faculty during the summer workshops and 
during the course development process. We plan to survey students about topics they are interested in, investigate how to 
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bringstudentsandfacultytogethertodevelop offerings around student-generated topics, and hold several advising sessions 

to improve faculty awareness of SYE choices and to gain more information about specific scheduling difficulties. 

0 Learning. Students and alumni told us they learned more when SYEs gave them the opportunity to work with students from 
other disciplines; included plentyofdiscussion, especially from their own and other disciplinary points of view; had time to 
reflect; and provided lots of active, real, hands-on experience. They also noted course management problems associated with 
teamteaching.Asaresult,wehaveaddedasecondsummerworkshoponteachingSYEthat includes discussion and exercises 

related to team teaching, active pedagogy, and student learning. We plan to work more closely with Otterbein’s new Center 
for Teaching and Learning to provide even more support for faculty members who risk doing interdisciplinary team teaching 
in a general education setting. 

The assessment committee suggested that we should collect more direct evidence of student learning; Karl Schilling noted 
the opportunity SYE provides for external critique; and a student member of the SYE committee suggested devising a brief 
generic pre-SYE and post-SYE survey that would measure the progress made in SYE by both students and faculty. We plan to 
ask faculty members to collect copies of projects, papers, journals, and student self-assessments for assessment purposes; 
to investigate the possibility of using SYE portfolios; and to encourage faculty members to hold public presentations of group 
projects to which the campus and external evaluators are invited. And we plan to follow up on the student’s suggestion. 

0 Teaching. Karl Schilling and faculty members told us that teaching SYE is a powerful faculty development tool; teaching in a 
team provides numerous opportunities for discussion, learning, and growth. But faculty members have also indicated that 
they need more support prior to, during, and after teaching in the program. As a result, we have added components on team 
teaching to the course development workshop and added a teaching SYE workshop to our summer program. We plan to make 
observation of offerings, formative discussions with SYE faculty, and faculty focus groups and surveys a regular part of the 
committee’s duties. We also plan to askSYE faculty members to write brief reflective essays about theirSYEteachinga couple 
of months after they have done so; these essays will provide us with information about how to help future SYE faculty and will 
provide faculty members themselves with a rich resource. 

0 Assessment. Sandra Fritz’s analysis and our observation told us that our interviewing of seniors needed to become more 
uniform. We plan to ask Dr. Fritz to hold a training session that covers not only our interview prompt and the interviewing 
process, but also coding tapes according to a rubric. 

The assessment committee and our experience ofworking with the data suggested that we needed to restate our program and 

offeringgoalsasexpectationsforstudentlearning,reviseourassessmenttoolstoalignwiththosechanges,andcommunicate 

the new goals more widely. We have clarified SYE expectations for both students and faculty, will revise our assessment tools 
accordingly, and will ask faculty to include the list of these expectations on their syllabi. Students in all SYE offerings can 
expect to 

o Analyze and engage with complex contemporary issues for which the liberal arts graduate is expected, by society, 

to provide leadership 

o Fractice the skills ofsynthesis by drawing from multiple disciplinary bases and by developing a “big picture” from 
diverse perspectives 

o Use interdisciplinary materials and methods and begin to recognize the strengths and limits of their own and other 
disciplines 

o Fractice active self-reflection, giving conscious attention to ethics, personal values, and citizenship, includingtheir 

roles and responsibilities as educated persons, professionals, and citizens 

o Practice the integration of learning, bringing together what has been learned in core courses and in major and 
elective courses to confront a contemporary issue 

o Reflect on liberal arts education, its purpose, and its uses 

Our assessment efforts have also told us what assessment efforts we need to make in the future. We need to continue what 
we’re doing, with some revision, and also add the following. 

o Assessment of spring-term juniors to gain better input data (lEO assessment model) 

o Regular alumni surveys, perhaps with a view toward a longitudinal study of SYE’s impact 

o A regular review process of all SYE offerings to ensure that SYE goals are being met, engaging pedagogy is being 

used, and documentation of student learning and achievement is occurring 
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o Amechanismforclosingtheassessmentloop-howwecanuseSYEassessment data to more effectively assess the 
college, its curriculum, and its support for student learning, and then effect change 

0 College. Working with seniors and faculty, listening to seniors and alumni, and analyzing our assessment data have taught 
the SYE committee a great deal, not just about the SYE program, but also about the senioryear in general. As a result, SYE has 
evolved into a program with a stronger emphasis on 

o Using one's whole education in the practice of citizenship 

o Reframing what one knows within a new context and in order to confront a complicated issue 
o Using the disciplinary expertise of students in an interdisciplinary setting 
o Having students use what they know and learn for something larger 
o Having faculty become partners with students in the face of complex, real issues 

o Learning how, not just what, through problem-solving, negotiation, doing, applying, interacting with the world 
beyond Otterbein 

o Seeing the senior year as a bridge to becoming an educated person, professional, and citizen 

SYE and its assessment have given us a clearer picture of seniors as they come in and leave SYE. Now that the college has this 
program as its endpoint for a baccalaureate degree, what can it tell us about seniors, about student learning, about 
Otterbein’s curriculum, about the college? What should we be doing at the freshman, sophomore, and junior levels to better 
prepare students for SYE and its purpose? What new questions should we ask about preparing students for SYE, preparing 
them for life beyond the college, and preparingthem for lifelong learning?Whatnewquestions should we askabout advising, 
about the balance of the major and the liberal arts core, and about the value of working with the major in contexts outside 
the major? What will we learn from public presentations, from external evaluators, and from portfolios? How can SYE help us 
close the assessment loop? 

We have learned the value of assessment for SYE, and we have learned the value of having multiple and specific assessment 
tools. We have learned what a powerful assessment tool the senior exit interview is for assessing student learning and for 
assessinga program. Might thecollege use the senior exit interviewto assess student learningonawiderscaleandtoassess 
itself? How might we revise course evaluation forms to include more tailored course evaluation questions that get at student 
learning? How might we enlarge the concept of self-assessment and portfolios to include the entire college career rather than 
one segment of it? 



Lessons Learned 

When we began theSYE pilot program, we suspected that simply asking seniors certain questions, both directly and indirectlythrough 
SYE, would cause them to reflect in a more metacognitive way on their entire education and thus frame their college achievements 
differently and see the point of their liberal arts study sooner. We now realize that in asking ourselves certain questions, both directly 
and indirectly through SYE. we are reflecting on the entire curriculum at Otterbein in a more holistic way and thus framing both SYE 
and an Otterbein education differently and seeing the point of seniors’ struggles more clearly. We have discovered that (1) collecting 
assessment data for external reasons and really looking at what that data say leads to change and the collection of assessment data 
for internal reasons-for making further Improvements; (2) having various kinds of data all tied to program goals provides the most 
persuasiveargumentsforkeepinga program and the most reliable suggestionsforstrengtheninga program; and (3) trying to measure 
where students arealways shows us where we are as well. We cannot be sure where all these questions will lead at this point, but given 
our experience of assessing the SYE requirement, we expect the recursive process to continue and the ripple effect to widen. 



Beth Rigel Dougherty is Professor of English and SYE Coordinator ot Otterbein College in Westerville, Ohio. 

Don Thompson is Associate Dean for Academic Affairs and Director of Institutional Research ot Otterbein College in Westerville, Ohio. 
Susan Thompson is Associate Professor and Choir, Mathematical Sciences Deportment, ot Otterbein College in Westerville, Ohio. 
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Active Learning: A Rubric Definition 

Cy Leise 



Introduction 

Active learning is a widely discussed concept that needs better operationalization. A simple Internet search results in a huge number 
of hits-even allowing for extensive redundancy and overlap-that leave one with a multitude of definitions and practices that are 
difficult to integrate. Two conclusions seem difficult to avoid: (i) a useful definition of active learning must be multidimensional, and 
(2) the educational processes required to implement active learning must be conceptualized in terms of a teaching/learning system. 
The first point, the focus of this paper, will be addressed by describing two new rubrics developed for measuring the level of active 
learning from the perspectives of learner and instructor. The second point will be addressed more briefly by suggesting some program 
assessment steps designed to produce outcomes associated with active learning. 



Resources for Rubric Content 

Many authors (e.g., Browne & Keeley, 2001; Loacker, 2000; Angelo &Cross, 1993; Centra, 1993; Schon, 1987) address the topic of active 
learning either directly or in relation to topics like assessment and/or evaluation ofstudent learningand offaculty teaching practices. 
Avery large number of Internet sites (see the Additional Resources at the end of this paper) provide definitions, links, techniques, 
simulations, handouts, procedures, lesson plans, measures, and other material to help instructors implement active learning 
practices in their courses. Many thoughtful insights and procedures are available in articles in association publications and 
proceedings (e.g.. Chambers & Parks, 2002; Higher Learning Commission, 2002; Jeffers, 2002). The challenge involved in adapting 
others’ practices into one’s own is a substantial barrier to rapid change, but clearly the rate of engagement is rapidly increasing if the 
number and quality of resources is a guide. 

By reading and noting how various authors labeled, used, or designed active learning practices, an extensive list of features and 
characteristics was gathered as a first step in conceptualizing what specific elements seemed significant for students and faculty at 
Bellevue University. This resource list plus extensive personal experience with systematically implementing active learning in 
introduction to psychology classes (Leise, 2002) provided a foundation for proceeding with a definition of active learning and the 
creation of the present rubrics. 



Rationale for Active Learning Rubrics 

Aclear definition is essential to creating a practical frame of reference for identification of learning performances considered “active” 
and to guiding efforts to improve. A definition also must support effective measurement of active learning outcomes. At Bellevue 
University we must meet this challenge because we have committed to implementation of active learning as central to our mission. 
Therefore, it is one of our institutional areas of focus approved by the Higher Learning Commission for our AQIP accreditation model. 

After I wrote a definition of active learning and began conceptualizing how to build an active learning rubric to address the 
measurement problem, it became apparent that there are significant differences of perspective between learners and instructors. 
Therefore 1 developed the two parallel rubrics discussed here. The first is a scale for rating professional progress of instructors in 
actuallycreatingactivelearningenvironments.Thesecondisascaleforratingpersonalgrowthofstudents/learnersinabilityto assess 
their own learning. Focus on instructor strategies is equal in importance to learner outcomes because active learning will occur only 
to the extent it is modeled, supported, and facilitated by instructors. Furthermore, the likelihood of student buy-in increases if 
instructors across disciplines consistently implement active learning strategies. These holistic rubrics can be used for any instruc- 
tional context or discipline for general program assessment. It is expected that more specific measures will be developed within 
courses and programs as data become available to guide more detailed interventions. 



Holistic Rubric Design Methodology 

Bargainnier (2003) provides an overview of types and purposes of rubrics, which include task-specific rubrics (e.g., to assess athletic 
performances); generic rubrics (e.g., to assess across-context oral communication performances); and holistic rubrics, which are 
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usefulforcomplexprocesses. Active learningfitstheholisticmbricmodel becauseitis composed ofmanycomponent processes, each 
of which could be separately rated or scored. Bargainnier lists the characteristics of useful rubrics: They must have clear performance 
criteria and rich descriptive language, focus on positive attainments, distinguish between performance and effort, and be reliable and 
valid. Apple and Krumsieg's (2001) “Methodology for Creating Performance Measures,” presented in Pacific Crest’s Curriculum Design 
Handbook, was the model used to develop the holistic active learning rubrics described here. 



Development of the Instructor Rubric 

Steps in the Apple and Kmmsieg (2001) methodology resulted in the following “set-up” information for the instructor rubric. (A copy 
of the rubrics is available from the author by e-mail: cleise@bellevue.edu.) 

□ Definition of Active Learning (Instructor Perspective) 

The following definition, and the one for the student active learning perspective (below), resulted from integration of common- 
alities across resources in the reference list. The wording intentionally presents a “profile” of effective instructors as a way to 
define active learning in a way that can be implemented, assessed, and measured. 

Instructors who use active learning strategies have positive attitudes about students’ com petencies, emphasize learning of effective 
processes with potential to generalize across areas of knowledge, design learning activities that can move learners step-by-step to 
higher-order levels of knowledge, and incorporate assessment into all aspects of the learning environment. Further, they strive to 
involve students by gainingtheir commitment to learn at a challenging level, by maintaining a warm social environment, by providing 
realistic tools and resources, and by helping them learn to transform any source of feedback into useful information for assessing 
performance. 

□ Ten Key Factors That Control Quality in Use of Active Learning Methods by Instructors 

The definition of active learning from the instructor perspective guided a brainstorming process that was summarized as ten 
factors (variables) judged to account for most of the differences among instructors in their implementation. 

1. Design courses and classroom activities around learning objectives and criteria of success. 

2. Use open or transparent evaluation systems focused on level of learning expected. 

3. Facilitate improved attitudes toward learning by helping students learn to use many sources of assessment feedback. 

4. Facilitate learning by means of interventions that encourage student ownership of the learning process. 

5. Integrate assessment of learning processes into courses to help students improve their outcomes. 

6. Adjust instructional methods and strategies to learning style and personality differences. 

7. Provide a variety of tools to support learners’ success. 

8. Support learning through methodologies and models that map main elements of complex assignments. 

9. Assume that learning at higher levels requires solid performance of lower-level skills. 

10. Assume that learning should transfer and generalize to significant life contexts. 



□ Five Integrated Factors for the Instructor Rubric 

The ten factors were further integrated by reducing them to five factors by creating statements based on pairs of closely related 
items. (These are ranked and paired in the list above; items i and 2 are the input for factor “a” below, etc.) This reduction made 
it possible for the final measure to capture the main variables that make a difference in active learning, while also keeping the 
amount of detail manageable. For the instructor rubric, the five factor statements that resulted were the following. (“Shorthand” 
phrases in bold print provide a quick way to reference the factors.) 




Factor a: Curriculum design. Curriculum outcomes are judged on the basis of standards that are related to reliably 
observable behavior and/or to measures. 

Factor b: Learning facilitation. Facilitation results in increased self-efficacy (accurate confidence) of learners because of 
their openness to any source of performance feedback. 

Factor c: Leveraging of learning styles. Integration of assessment of learning processes results in higher quality, and 
enriched outcomes through “leveraging” variations in individual learner goals, styles, and personality. 
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Factor d: Variation of resources. Resources, tools, and models for learners are selected to support their success with 
complex challenges. 

Factor e: Learning generalization. Generalization of learning is achieved by focus on building higher-level skills needed 
in life contexts. 



□ Labels for Five Levels of Quality of Instructor Facilitation of Active Learning 

Having established the five factors (variables) that “anchor’’ instructor performance related to facilitation of active learning, five 
levels of competency were conceptualized and labeled. Because performance in this complex area involves long-term growth, 
the top levels describe depth and breadth of expertise that is unusual. Some instructors find it professionally beneficial to put 
more intense energy into discipline-based research; others may find teaching/learning scholarship and research more suitable 
for their career progress. The instructor rubric is designed to indicate that the middle level, level 3, is considered strong 
performance-a level to be expected in a program assessment system. The intent of the levels is to clearly identify performance 
that is below expectations as well as performance levels to be aspired to. 



5. Transformational 
4. Enriching 
3. Engaging 
2. Conventional 

1. Risk Averse 



Uses learning challenges and facilitation methods that result in life change in learners. 

Moves learners to explore knowledge applications beyond expectations. 

Consistently produces active learners who can self-assess performance against standards. 

Manages learning by clearly defining outcomes; discounts the value of facilitation of learning 
processes. 

Avoids change; discounts the value of assessment of teaching strategies or of supporting learning 
processes. 



□ Final Format of the Instructor Rubric 

Having established the five factors and the five levels, items were written to reflect how each factor is most likely to be 
demonstrated at each level. Thus factor a at level 1 (Risk Averse) indicates instructor behavior and attitudes related to restriction 
of learning caused by setting objectives related only to content in a specific course text. Factor a at level 3 (Engaging) indicates 
much stronger curriculum design including clear standards of performance. 

The goal for the set of items across factors a through e at each of the five levels is to create a gestalt, or coherence, in the total 
picture at each level. Thus at level 3, all items should be consistent with each other in the sense that it is very likely that someone 
who creates curricula in the way described will also support accurate self-assessment skills in learners. 



Creation of the Student Rubric 

The Student rubric was developed to be parallel to the instructor rubric. The student behaviors described at each level are assumed 
to be related to, if not caused by, instructor active learning facilitation at the analogous level on the instructor rubric. 

□ Definition of Active Learning (Student Perspective) 

Students committed to active learning have positive attitudes about their own competencies and accept the larger goal of learning- 
to-learn so they can become highly effective at using processes with potential to generalize to work, life, and citizenship. They 
appreciate well-designed learning opportunities that help them move step-by-step to higher-order levels of knowledge. They have 
learned the affective skills involved in changing their attitudes about assessment, with the result that they can transform even 
negative, evaluative feedback into useful performance feedback. They strive to set their own learning standards above those 
required to ensure that they continue to challenge themselves. They maintain a positive social environment with peers and 
instructors by continually improving their communication and teamwork skills. They learn to continually review the tools and 
resources provided by the instructor and seek out additional ones that have potential to maximize performance. 

□ Ten Key Factors That Control Variability in Use of Active Learning by Students 

1. Commit to well-designed curricula that support learning steps and include clearly defined outcomes and standards. 

2. Question the connections between the learning processes and the outcomes to assure themselves that the evaluation 
system is transparent. 

3. Expand openness to assessment by overcoming performance anxiety and frustration. 

4. Achieve a sense of ownership of the learning process through learning from instructor facilitation. 







io6 /A Collection of Papers on Self-Study and Institutional Improvement, 2003 - Volume 3: Promoting Student Learning and Effective Teaching 



5. Respond to the full implications of instructor and peer assessment feedback by continuing to use insights across all 
learning contexts. 

6. Adjust strategies for achieving performance standards by taking into account individual learning style and personality 
differences. 

7. Use all relevant resources and tools, including those previously acquired, to support success with learning processes 
related to outcomes. 

8. Break down complex tasks into steps by using methodologies and models that guide or map how the main elements 
should be addressed. 

9. Assume that learning at higher levels requires solid performance of lower-level skills 

10. Assume that learning should transfer and generalize to significant life contexts. 



□ Five Integrated Factors for the Student Rubric 

As was done with the instructor rubric, ten factors resulting from a brainstorming process for the student perspective were 
“integrated” by creating five factor statements based on pairs of closely related items. (These are ranked and paired in the list 
above; items 1 and 2 are the input for factor “a” below, etc.) This reduction made it possible for the final measure to capture the 
main variables that make a difference in active learning for students, while also keeping the amount of detail manageable. The 
five factor statements that resulted were the following. (The shorthand phrases in bold print provide a quick way to reference the 
factors.) 

Factor a: Commitment to curriculum. Commit to well-designed, challenging curricula judged on the basis of clear 
standards that are related to essential learning processes. 

Factor b: Learning to learn. Achieve self-confident ownership of learning by effectively using assessment to learn how to 
learn. 

Factor c: Leveraging of learning style. Develop learning attitudes and strategies that enhance ability to overcome blocks 
to learning and that make strengths of individual learning style and personality traits. 

Factor d: Varied use of resources. Applyinformationprocessingskillstoselectresources,tools,andmodelsthat are most 
relevant to addressing complex challenges. 

Factor e: Learning generalization. Achieve generalization of learning by building higher-level skills needed in life con- 
texts. 



□ Labels for the Five Levels of Quality of Student Engagement in Active Learning 



5. Transformed Learner 
4. Enriched Learner 

3. Engaged Learner 
2. Conventional Learner 
1. Risk-Averse Learner 



Sets personal challenges that result in significant life change and growth. 

Actively explores howto use knowledge for applications of personal interest; performs beyond 
expectations. 

Active learner who can self-assess performance against standards. 

Manages learning by focusing on outcomes; limited in awareness of personal potential. 
Grade-oriented; concerned about meeting requirements efficiently. 



Program Assessment Considerations 

The following steps, based on experience at Bellevue University, illustrate one approach to incorporating active learning into 
institutional program assessment. 

1. Provide professional growth experiences to introduce active learning principles and the rubrics. 

2. Request reflective reports by individual faculty members on the state of active learning strategies in one of their courses. 

3. Hold a faculty interactive workshop to assess progress and plans for all program assessment projects. 

4. Request each faculty member to assess active learning in a few student cases to identify variations. 

5. Present program data for percentage of courses that meet the level 3 standard in the instructor rubric. 
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6. Present program data for percentage of students who perform at level 3 on the student rubric. 

7. Continue annual data collection, workshops, and reports to increase quality and quantity of implementation. 

Summary and Conclusions 

Active learning assessment will not improve unless clear definition and measurement foundations are set in operational terms. 
Information resources and a rationale are offered for two multi-dimensional rubrics— instructor and student perspectives. Detailed 
information is presented for steps in the design methodology. The recommended level of performance is level 3 of five levels on each 
rubric. Levels 1 and 2 identify performance that is below expectations; levels 4 and 5 represent performance quality that will require 
longer periods oftime as the result of personal and professional growth. Future challenges are suggested in the implementation sequence 
planned at Bellevue University. The most significant function of the described rubrics is to create a definite operational starting point that 
isflexible across programs and that is an “open” guidefor creative options. As with any tool, successful use will depend upon commitment 
to continuous improvement on the basis of empirical information and reflection on teaching/learning processes. 
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Additional Resources 

The following sites, plus many more, are easily accessed by searching with the term “active learning.” 

Campbell, K. 2003. The Web: Design for active learning. Provides an overview of a constructivist approach to active learning: makes 
clear connections to educational theories. 

Dodge, G. 2003. Active learning on the Web. Internet site on using the Web for student research. 

Felder, R. 2003. Active learning and cooperative learning. Emphasizes the opportunities for active learning through teams. 

Fink, L. D. 2003. Active learning. Presents a model of active learning involving four components: two kinds of experience— (observing 
and doing), and two kinds of dialogue-(with self and with others). 

Fleming, N. D., and C. C. Bonwell. 2003. VARK: A guide to learning styles. 

Harvard Project Zero. 2003. ALPS: Active learning practices for schools. 
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National Teaching and Learning Forum. 2003. Provides active learning definitions and describes practices. 

Sage Journals Online. 2003. Provides information for Active Learning in Higher Education, an electronic journal available by 
subscription. 
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English 101 Writing Assessment That Keeps 
Both the **Quants” and the **Quals” Happy 



Introduction 

The English department at College of Du Page, after several years of work, instituted mandatory placement for students enrolling in 
English composition courses. The program became operational in fall 2001 as a project of AQIP. In keepingwith AQIPgoals, we needed 
to measure the success of the students who had been placed in developmental courses as they left their developmental course(s) and 
also as they completed their first course for credit-English 101. We had to compare the developmental students leaving English 101 
to their peers who had not been required to take developmental English. 

With more than five thousand students enrolled in English 101 each year, this promised to be a major project involving the entire 
English faculty. It seemed like a good idea to expand the purposes of the assessment to include English 101 program review. To do this, 
we designed an assessment that could serve the purposes of comparing developmental students with students who had not been 
placed in developmental courses while at the same time giving us qualitative information about 101 student writing that we could use 
to evaluate our program with the goal of improvement. 



Baseline Data and Assessment Design 

To evaluate to what extent mandatory placement is effective in improving student writing, we needed to measure student writing 
before mandatory placementwas instituted. Fortuitously, the English department had collected data on student writingin springand 
falli 997 andwinteri 998 ,severalyears before mandatory placement was instituted. There was, however, mandatory testing. Students 
scoring below the cutoff point were advised, but not required, to take a developmental writing class. Some students followed this 
advice; many did not. At that time students at the end of English loi had written essays ofapproximately two typed pages in response 
to a prompt based upon an article. After consulting with Harlan Schweer and Prudy Widlakofthe research and planning office, who 
were available for advice on constructing an accurate and effective assessment instrument, we decided to use these essays to 
establish a baseline. In addition, we consulted them about evaluatingthe essays by holistically groupingthem in quartiles, a plan they 
accepted. Presentation of the idea to the English faculty raised questions and comments resulting in explanatory documents 
(Appendix) that allayed many of the initial misgivings and answered some of the questions. 



Data Collection 

Through consultation with the office of research and planning and a series of meetings with faculty, we determined the precise form 
of the essays for our post-mandatory placement assessment, which needed to be similar, but not identical, to the baseline 
assessment. In the fall quarter of 2002, instructors were asked to turn in the completed student assessment essays during the week 
of December 10-17. We received essays from approximately 92 percent of the English 101 classes, which pleased us. 



Analyzing Baseline Data 

The day before fall 2002 quarter classes began, the full-time faculty met to analyze the baseline data collected in 1997-1998. Each 
faculty member was given a twenty essays and was instructed to read the essays holistically and ranktheminfour equal-sized groups. 
Previously, the set of essays had been copied onto three different colors of paper so we could readily recognize duplicate copies. The 
sets were shuffled systematically so thatfour different readers read individual essays and each read a different combination of essays. 
We encouraged the faculty to finish reading in less than an hour and most did so. During lunch, the assessment team and other 
volunteers worked to sort the essays into quartiles. If three or four raters judged an essay as belonging in a particular quartile, it was 
included in that quartile for the quartile descriptions. The other essays were not used for quartile descriptions. 



Lynn Sykes and Helen Szymanski 



ERIC 




no/ Collection of Papers on Self-Study and Institutional Improvement, 2003 - Volumes: Promoting Student Learning and Effective Teaching 



As might be expected, there was more agreement on essays in the highest and lowest quartiles (nineteen and eighteen, respectively) 
than there was on essays for the two middle quartiles (nine for the second quartile and seven for the third quartile). Raters reported 
that they had more difficulty determining where the middle essays should be placed. Out of two hundred essays read, three or four 
raters agreed on the placement of fifty-three essays. This should not be taken to mean that the raters had widely different standards. 
Rather, this distillation is part of the design of the assessment. By shuffling the essays for each reader, we ensured that each essay 
would be read in a different context. Since the readers were instructed to have equal groups, essays would be ranked differently 
depending on the other essays in the set. Thus, essays that consistently ranked in the same quartile can be considered to be 
representative of that quartile, rather than borderline cases. 

In the afternoon, the faculty reassembled. Each table analyzed the characteristics of the essays belonging in one quartile and then 
compared them to essays in the adjacent quartile(s). Since there were four quartiles and eight tables, two tables worked on each 
quartile, at first separately, then together. Most tables were finished with their descriptions in an hour and a half. Later, we typed the 
quartile descriptions, combining and organizing them. These quartile descriptions represent the qualitative data. We now have a 
description of the writing characteristics of English 101 students at the end of their course. These data are useful for evaluating the 
first composition course, seeing overall strengths and weaknesses, and making decisions about areas on which to focus. 

We simultaneously collected quantitative data on students who had been recommended for remediation. By identifying those 
students, we were able to see in which quartile they placed. Of the fifty-three essays chosen for the final quartiles, eight were written 
by students who had been directed to a developmental course. Five of those were placed in the lowest quartile. The writer of one of 
those five essays had taken a developmental writing course, as advised. The others did not take a developmental writing course. Two 
other students whose essays ranked in the lowest quartile had not taken the placement test. 



Table 1 

Quartile Distribution of Students Advised to Take Remedial Courses 



Quartile 


1 (Top) 


2 


3 


4 (Bottom) 


Total 


19 


9 


7 


18 


Advised to remediate 


2 


1 


0 


5 


Remediated 


1 


1 


0 


1 


No placement test 


0 


0 


0 


2 



Raters ranked one essay from a student who had been recommended to take a developmental course in the second quartile. That 
student had taken developmental reading and writing, as advised. Two of the essays were placed in the top quartile. The writer of one 
had taken developmental writing. The other student did not take anydevelopmentalcourses. In addition, her reading score was quite 
low, 017, mysterious, or, should I say, suspicious? 

Initially, we did not intend to collect quantitative data for this set of essays, since mandatory placement had not been instituted. We 
do plan to collect such data when we evaluatethe post-mandatory-placement data we are collectingthisyear.Atthattime, we will need 
a larger sample of essays from developmental students. We can get that by assigning an average grade to each essay by a former 
developmental student that we evaluate. Then, even if the essay is not used for the qualitative quartile descriptions, it will be scored. 
For example, if it is ranked in the third quartile by two raters and the fourth by the other two, it will be scored as 3.5. Then we will have 
a quantitative assessment of the developmental students as they complete 101. 



Implementing Assessment Data 

We used our in-service day to begin the process of implementing what we learned from our baseline assessment. Full-time English 
faculty met off campustodiscuss the quartile descriptions and whatthey indicated weshouldworkonoverthecomingyear. The faculty 
arrived at the following themes: 

o Thinking made visible: critical and creative thinking 
o Content and design: creation and organization beyond formulas 

o Self-awareness, self-evaluation, and style: seeing oneself as potentially able to negotiate and alter the world 



These themes will form the basis of in-house conferences or workshops at which faculty can learn from others and share ideas with 
one another. They submitted ideas for a teaching anthology to be distributed among the faculty and to be used as a resource for 
conference planners. 
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We established dates for a winter conference and a spring conference, and we plan to have one in the fall as well. These conferences 
will occupy an afternoon and be open to full-time and part-time English faculty. Forthe winterconference, we engaged a speakerfrom 
our philosophy department to make a presentation on critical thinking, and several faculty members, selected by reviewing the 
teaching anthology for pertinent themes, agreed to prepare poster presentations of their ideas for encouraging critical and creative 
thinking in their own classes. 



Lynn Sykes is Associate Professor at the College ofDuPage in Glen Ellyn, Illinois. 

Helen Szymanski is Assistant Professor at the College ofDuPage in Glen Ellyn, Illinois. 
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Appendix 

Holistic Quartile Grouping 



Haw is holistic quartile grouping done? 

Readers each receive a bundle of about twenty orthirty essays. They read each essay holistically, judging its overall quality as a response 
to the essay prompt. After readingan essay, they place the essay in one of four personal groups, rangingfrom the best essays to the worst. 
As readers make their way through their bundle of essays, they adjust the growing groups so they end up with four equal groups. 

Haw long will the reading take? 

It usually takes thirty to forty-five minutes to holistically read and sort twenty-five or thirty essays, but we will allow an hour. 

Equal groups? Suppose the essays da not divide equally? 

It is likely that most bundles of essays will not divide equally. Readers need to remind themselves that they are not assigning grades 
to the essays, but simply saying, “These six (or five, or seven) essays were the best (second ranking, third ranking, or worst) in my 
bundle.” When all the readers assign essays to one of four equal groups, we eliminate one common variable-readers judging essays 
more or less stringently— and thus make our evaluation more uniform. 

But what if my bundle is particularly bad? Then my “best” group will actually be only “best of the worst." 

True. Rrst, we need to remember that we are not grading individual students. If you have a “best of the worst” group, someone else will 
likelyhavea“worstofthebest”group,sothenumberswillbalanceout.Second,thesameessaysyouarereadingaresimultaneously being 
read by other people in the context of different bundles. After all the groups are combined into a quartile, if the majority of readers do not 
agree on an essay’s placement, it will be regarded as ambiguous and will not be considered in the quartile description. 

What happens to the groups when the readers are pnished? 

The groups will be consolidated into super groups-quartiles. The assessment team will sort through the essays in the quartiles. Only 
those essays that show up consistently in the quartile will be retained. 



Explanation of Quartile Description 

Haw will we go about describing the quartiles? 

After the essays have been grouped into quartiles and the ambiguous essays have been removed, the faculty will divide into groups 
to develop a description of the essays in a particular quartile. To do this, they will read over the essays to find commonalties among 
them. In addition to describingwhat the essays in their quartile have in common, theywill give examples of each trait that they notice. 

Won’t the essays in any one quartile represent a range of writing skills? 

Probably they will. The description needs to account for this range and give examples ofthe variety that shows up in that quartile. 

That sounds very open-ended. 

It is, but one way the description will be limited is through contrast with the adjacent quartile. Each group will need to look over the 
essays in an adjacent quartile and describe the ways the essays in the top quartile, for instance, differ from those in the second 
quartile. Another limitingfactorwill be thecategories we traditionally use to describewriting(controllingidea, evidence, organization, 
style, mechanics, and so on). 

What can we learn from this description? 

The description will give us particulars about our students’ writing that cannot be found in a numerical score (quantitative analysis). 
With information about essay organization-for instance, including examples of organizational strategies (or the lack thereoO that 
typify each quartile-we will be able to identify those areas that fail to meet our expectations. Armed with this information, we can 
address our composition program, from active course files to staff development, more effectively. 
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Creating and Implementing a 
More Focused and Effective 
General Education Program 



John C. Catau, William H. Cheek, and James P. Baker 



Background 

Southwest Missouri State University (SMSU) is a comprehensive, metropolitan university with an enrollment of more than 20,000. 
Prior to the implementation of its new general education program in fall 1997, the SMSU program was an unwieldy and poorly focused 
conglomeration of more than 350 classes. Because the aims and goals of this program were so vague and poorly defined, the menu 
of available classes included numerous courses designed more to introduce potential majors to a specific discipline than to 
accomplish a specific set of general education goals. In some instances, virtually all courses in an area (e.g., philosophy, literature, 
religious studies) were among the many choices available to students. With no permanent general education committee, oversight 
of the program was a responsibility of the entire faculty senate. Efforts to reform the program inevitably led to turf wars that resulted 
in many more additions than deletions. 

The results of several unsuccessful attempts at comprehensive general education reform were obvious to North Central Association 
accreditation site teams. The 1985 site team report, for instance, included the following concern and suggestion: “The general 
education requirements do not match the institution’s self image. The frequent revisions have left the university with reiterations of 
the ‘cafeteria’ system without achieving general education. The team suggests a major effort before the next decennial review to 
develop a coherentcomponentofgeneral education appropriate to a universitycommitted toexcellence in undergraduate education.” 



Reform Efforts 

The faculty senate responded to the 1985 site team report by establishing a committee charged with reviewing and making 
recommendations for change in the university’s general education requirements. Although the committee proposed a reformulation 
of the general education requirements in 1987, the senate was once again unable to reach a consensus, and the reform effort failed. 

The next attempt to generate comprehensive general education reform began in 1993 when, shortly after his arrival on campus. 
President John Keiser indicated his strong support for change by appointing a select general education reform committee. Composed 
of highly respected membersoftheuniversityfaculty, this committee was charged “toassessthecurrent program of general education 
and to develop a new program that will prepare students for their academic careers as well as for their tasks as citizens in the 21 ^' 
Century.” The new program was to be “coherent, connected to the overall mission of the university, capable of continuing and lively 
development, and competently managed.” 

The committee began work with the understanding that, in light of its charge as well as the 1985 site team report, cosmetic changes 
to the existing general education program would not suffice. This perspective became even more certain when the university’s efforts 
to redefine its mission culminated in a new statewide mission in public affairs. The goal of this new mission, which was signed into 
law by Governor Mel Carnahan in June 1995, is “to produce citizens of enhanced character, more sensitive to the needs of community, 
more competent and committed in their ability to contribute to society, and more civil in their habits of thought, speech and action.” 
As a university-wide concern, the public affairs theme is to be “housed and implemented in all units and departments.” From an 
academic perspective, the public affairs mission means that “in addition to mastering one or more disciplines enabling problem 
solving in a variety of occupational settings, the student is understood as also having a life-long career as a citizen, for which discrete 
competencies and moral commitments are necessary.” General education linkages tothisnewmission became an element ofconcern 
for the reform committee. 

After nearly two years of intensive work including several open forums to encourage faculty input, the general education reform 
committee offered its new vision for general education to the 1994-1995 faculty senate for consideration and action. Consistent with 
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the new public affairs mission, the overriding aim of the new program “is to develop people capable of making thoughtful choices that 
lead to creative and productive lives and to responsible participation in society.” In a marked departure from the earlier general 
education program, the new program included a precise and comprehensive inventory of goals for learning in general education. It 
declared that “general education is that portion of the curriculum in which the university directly addresses the knowledge, skills, 
habits, and dispositions of educated persons...(lt) provides for learning that educated persons will use throughout their lives in their 
many roles and communities.” The specific goals of the program were divided into two subcategories: (1) those pertaining to 
intellectual abilities and dispositions and (2) those pertaining to knowledge and understanding. 

With the aims and goals of the new program in place, the faculty senate debated and approved a framework for the new general 
education curriculum. Instead of identifying the specific classes that would comprise the program, the senate merely identified the 
overall structure of the program. It agreed, for instance, that the basic required courses portion of the new program would include 
classes designed to help the students develop and/or fine-tune their skills in computer literacy, public speaking, writing, mathemat- 
ics, and health and well-being. The senate also agreed with the reform committee's recommendation that all newstudents be required 
to complete an introduction to university life class during their first semester on campus. The purpose of this latter class would be 
twofold: to improve student success (retention) while also Introducing all students to the public affairs mission. 

In recognition ofthe factthat“developingeducated people requires intensive study in many areas of inquiryaswellas interdisciplinary 
explorations,” the senate agreed that a second component ofthe new curriculum would require each student to complete classes in 
three areas of inquiry. Because a “study ofthe natural world provides an understanding of important principles and methodologies 
for making choices,” the first oftheseareaswould focus on developingan understanding ofthe natural world. The culture and society 
area of inquiry recognizes that “informed choices require knowledge of what people have done and imagined, currently as well as 
historically. [And] choices are also tempered by knowledge ofthe social and cultural settings in which they are made or were made 
In the past.” Classes from both a humanities and a social sciences perspective would be required in this area. The final area of inquiry, 
self-understanding, is based on the recognition that “to make informed choices, one must understand the natural and social context 
in which one lives and must heed the ancient injunction to ‘know thyself.'” In this case, th ree perspectives would be required: social/ 
behavioral, humanities, and creativity and vision. 

The final component ofthe general education program was designed to offer a direct link to the university's public affairs mission. It 
would be made up of two American studies classes (a survey of American history and “American Democracy and Citizenship”) as well 
as an upper-division capstone class entitled, “Public Affairs Issues for the 2i"‘ Century.” The latter class, which should not be taken 
until a student has completed a minimum of sixty credit hours. Is intended to integrate experiences generated by the other parts of 
the general education program while focusing on public issues and choices of broad Importance. 

In another significant development, the faculty senate also created a new permanent committee, the committee on general education 
and intercollegiate programs (CGEIP), to oversee and evaluate the general education program as specified by the Constitution and 
Bylaws ofthe Faculty. One ofthe first tasks this committee faced was the selection of all the classes that would comprise the new 
general education curriculum. None of the classes in the old program was automatically accepted Into the new program. Instead, 
during the 1995-1996 academic year, departments were asked to submit proposals for classes for each of the general education 
categories. As a part of these proposals, the departments were instructed to show how the suggested class would address the aims 
and goals ofthe program. Each proposal was also required to include an assessment plan designed to enable the faculty teachingthe 
class to evaluate its effectiveness as a general education class. Proposals receivingthe approval of CGEIPwere forwarded to the faculty 
senate for its approval. Successful proposals were then sent to the vice president for academic affairs and the university president 
for final approval. In the end, fewer than a hundred classes were approved for inclusion In the new general education program that 
was Implemented in fall 1997. 

While the structure ofthe new general education program was under debate in the faculty senate, and before the specific courses 
comprising the program were developed by departments and approved through the curricular process, the university experienced 
another NCA site visit. Not surprisingly, the 1995 site team identified the university's general education program as one of six concerns 
in its final report. Specifically, the team members noted that “while recent planning efforts in general education are encouraging, the 
same general education requirements are in place which were cited as a concern in 1985.” 



Implementation and Oversight 




In recognition of the fact that one of the weaknesses of the preceding general education program was a lack of continuity and 
administrative oversight, the vice president of academic affairs created a new position, the associate dean ofthe university college 
for general education, during summer 1995. This meant that the person selected for this position. Dr. )ohn Catau, who is also an 
ex officio member of CGEIP, was closely involved in the evaluation of all the general education course proposals. He was also in place 
when the new program was actually implemented in fall 1997. 
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Although curricular control of the general education program remains the prerogative of C6E1P, Dr. Catau oversees, coordinates, and 
administers all other facets of the general education program. Because nearly all of the general education classes are taught by the 
faculty in the departments that developed them. Dr. Catau works closely with those departments. The upper-division capstone class 
is a significant exception. “Public Affairs Issues for the 21 ®' Century” is a variable topics class designed to be writing- and discussion- 
intensive as well as multidisciplinary in its perspectives. Individual sections of the class must be developed by faculty from at least 
two different departments. Once a topic has been fully approved, it may be taught by any interested and qualified faculty members. 
Some sections are team-taught; most are not. Since these classes are not “owned” by i ndividual departments, the associate dean is 
responsible for their oversight. 

Astheuniversity'sarticulationofficer,the associate dean works closelywithothercolleges and universitiesto ensure that the transfer 
process to SMSU is as smooth as possible. He also serves as the primary appeal officer for those students with general education- 
related issues. The most common category of appeals is those made by students who believe their transfer credits have been 
incorrectly evaluated. Direct course equivalencies are created by department heads. In those instances when courses are not 
considered the equivalent of a specific SMSU class, however, the associate dean may assign a general education equivalency when 
it appears that the class meets the “spirit” of a specific general education requirement. 

In addition to its role in approving changes in the general education curriculum, C6E1P is also responsible for overseeing the quality 
of the program. According to faculty senate action, “The C6E1P also will periodically review all general education courses to see that 
they attempt to achieve the Goals of Learning listed in Part One: Intellectual Abilities and Dispositions of the General Education 
document. Courses in the Areas of Inquiry portions of the General Education program will be reviewed to see that they also attempt 
to achieve the appropriate Goals for Learning listed in Part Two: Knowledge and Understanding....Each general education course and 
each section of the Capstone course will be reviewed at least once every three years to determine if sufficient interest and enrollment 
exist to continue the course or Capstone section in the General Education Program.” 

With this responsibility in mind, the CGEIP has divided the general education program into three subcategories. Every year all of the 
classes within one of these subcategories are up for evaluation and review. After several false starts and considerable hesitancy, the 
CGEIP has agreed that as a part of this process, departments must submit an assessment portfolio consisting of 

o A written statement by the department head, program director, or appropriate designate that summarizes the status of the 
class 

o Copies of the course policy statements and syllabi for every section of the class taught in the most recent semester 

o A table that shows how the goals of the course under review are linked to the specific goals of the SMSU general education 

program 

o A copy of the assessment plan in effect at the time of the review (The CGElP*s interest in this element is limited to whether the 
department has an effective assessment plan in operation. The results of the assessment plan are not required in the 
portfolio.) 

o Enrollment data by semester for the course 

Based upon its reviewofth ese materials, theCGElPcan recommend thata class be allowed tocontinuein the program without change, 
be allowed to continue in the program with change(s) that must be made before the next review, or be removed from the program. 

In addition to the evaluation requirements of the CGEIP, the Center for Assessment and Instructional Support also contributes to the 
assessment of both individual courses and the overall general education program. The director of the center, Martha Kirker, assists 
the departmental faculty in developing their assessment plans. She is also available to help the faculty analyze, interpret, and use 
the assessment results to improve the effectiveness of their classes. Furthermore, Dr. Kirker administers and reports the results of 
a required general education exit exam. 



Conclusion 

After years of numerous unsuccessful attempts to reform its general education program, the faculty of SMSU responded to concerns 
expressed by NCA site teams and a clear commitment from a new university president with yet another attempt at comprehensive 
reform. A select committee of campus leaders began the task of designing a new general education program with the realization that, 
due to the size and complexity of the university, it would be impossible to completely avoid a distributional approach. With a fresh man 
class of more than two thousand students each year, small and nontraditional classes would be unrealistic. Despite these 
circumstances, the committee crafted a new program that was more focused, with many fewer course options, a direct link to the 
university’s new mission in public affairs, and a much improved oversight mechanism. None of the classes in the old program was 

Er|c 121 



iiB/k Collection of Papers on Self-Study ond Institutional Improvement, 2003 - Volume 3: Promoting Student Learning and Effective Teaching 



given a free pass Into the new program. To ensure that all the courses In the program address a specific set of general education goals, 
regular evaluations are conducted by a standing committee of the faculty senate. Contrary to past trends, the first five years of the 
program’s existence have resulted In few additions to the general education curriculum. Hopefully, the 2005 site visit team will be 
favorably Impressed by SMSU’s more focused and effective general education program. 



Notes 

All the quotations in this paper come from internal documents: 

Constitution and By-Laws of the Faculty 

Countdown to the SMSU Centennial: A Long Range Vision and Six- Year Plan (2000-2006) 
The General Education Program, Southwest Missouri State University 
NCA Report of a Visit to Southwest Missouri State (1985) 

NCA Report of a Visit to Southwest Missouri State University (1995) 

Welcoming the 2i*‘ Century: A Long Range Vision and Rve-Year Plan (1995-2000) 



John C Catau is Associate Dean of University College at Southwest Missouri State University in Springfield. 

William H. Cheek is Associate Dean of the College of Natural and Applied Sciences at Southwest Missouri State University in 
Springfield. 

James P. Baker is Executive Assistant to the President at Southwest Missouri State University in Springfield. 
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It Can Happen: 

General Education Reform 
Through Assessment 

Eric Gardner, Diane Boehm, Clifford Dome, and Mary Hedberg 



Introduction 

This session will detail Saginaw Valley State University's (SVSU) general education reform, which was spurred by a multi-layered 
assessment process. This paper offers the “who” and the “what”-the institutional context and the specifics of the assessment and 
reform processes. The session itself, while considering these issues, will focus more on the “how” and the “why” behind those 
processes. In this, through interactive discussion, we will invite session participants to place our work in dialogue with questions of 
general education, assessment, and curricular reform at their own institutions. 



Who, What, etc.: Institutional Context and the Need for Change 

SVSU is a regional comprehensive university with an enrollment of approximately 9000 students. It offers a range of bachelor's degree 
programs and a handful of master's degrees; though several of its students are transferees, it offers large and diverse programs for 
first- and second-year students. 

University faculty are unionized and maintain a strong voice in curricular planning and development. A curriculum committee 
composed of six elected faculty members, two administrative appointments, and one student serves as the first level in the curricular 
approval process. Their decisions are then subject to full faculty approval and administrative approval. 

Almost since its founding in 1963, SVSU has envisioned general education (GE) as a set of various content area requirements. Over 
time this conception progressed into a thirty-nine-credit-hour requirement spread over nine content categories-literature, arts, 
numerical understanding, natural sciences, historical and philosophical ideas, social science methodologies, social institutions, 
communication, and international perspectives. 

Courses were placed in general education through the above-described process, but there was no oversight body, no standard review 
and/or expiration policy for GEcourses, no statement of objectives forthe program (though individual categories did have objectives), 
no criteria forthe various categories, and no provision forongoingassessment. Further,therewere no regularinstitutional procedures 
in place to remove courses that were not servingthe goals of GE, short of the department in question requestingto remove the course. 
For the most part, this did not occur because departments had rather powerful vested interests in maintaining courses in the GE 
program. Similarly, because there was a perception that individual departments (or small groups of departments) “owned” 
categories, there was little argument over adding additional courses—as long as a department did so in “its” category only. Thus, the 
1990s saw a dizzying accumulation of more than 260 different courses in the program. 

In 1994, NCA evaluators critiqued the SVSU GE program and echoed concerns that had spread across campus: that general education 
at SVSU needed sweeping reform. Initially, this led the faculty to create a statement of their rationale behind general education and 
then to assess the program in terms of this rationale. 



Initial Assessment 




In response to the NCA evaluators' critique, in 1995 SVSU submitted a plan to NCAthat defined the steps we would take toanswerthe 
concerns raised and proposed a timetable for carrying out an assessment and revision of general education. One of the first things 
to be done was establish a committee to articulate SVSU's rationale for general education, defining what the institution wanted to 
accomplish through the general education program. In developing this mission statement, the committee proposed that general 
education should foster critical thinking, cognitive knowledge, and affective understanding along with oral and written communica- 
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tion skills. Once the faculty ratified the new rationale for general education, the question arose whether the general education 
program, with Its various categories and large number of courses, accomplished these newly defined goals. 

After considerable faculty^wide discussion, the faculty voted to declare a moratorium on adding or deleting courses from the general 
education categories for two years, while the program was assessed. A new committee was formed, which included administrators 
and faculty, to draft an assessment measure. Once an assessment process was agreed upon. Professor Hedberg was appointed to lead 
the general education assessment task force. The intent was for this process to include a large number of faculty from across the 
university. Including senioraswell as newerfaculty. For fourconsecutive semesters, facultytaskforces, composed ofdifferent individuals 
eachtime,althoughchairedbythesamefacultymember, presented large numbersofstudentswith “prompts” selected bythe committee. 

The committees Introduced these prompts, which included excerpts from films, book reviews, and analytical writing, to students in 
Introductory as well as upper-division classes and asked students to respond both orally and in writingto a series of broad questions. 
Task force members then assessed the responses according to rubrics developed for each prompt. Although faculty on these 
committees were selected from a number of disciplines and approached evaluating responses in various ways, the conclusions were 
similar across the four semesters: students were adept at acquiring new information but had difficulty thinking critically about that 
Information, applying it to new situations, seeing flaws in arguments based on that information, or recognizing rhetorical strategies 
used by authors and filmmakers to shape a viewpoint. All four task forces also noted that students at all levels demonstrated 
disappointing levels of oral and written communication skills. 

The final report, presented by the chair of the task force to both the administration and the faculty, concluded that general education, as 
itwascurrentlyconstituted,did not align with the rationale forgeneral education and was not effectively accomplishing its defined goals. 

Additional analysis showed that although there were large numbers of courses in each of the content categories that made up the 
general education program, most students fulfilled their general education requirements by taking a select number of these courses. 
In spite of the turf wars over courses approved for these categories, in actuality departments were not gaining significant numbers of 
students for most courses in the categories just because of their inclusion in one of the general education categories. By and large, 
courses had not been developed to accomplish general education program goals, and the courses that were attractingstudents were 
the introductory courses for majors. 



Program Development 

Building from the results of this initial assessment, SVSU embarked upon a major GE reform project requiring extensive strategic 
planning and consensus building among the faculty. Initially, an ad /?oc committee composed primarily of faculty was formed. Their 
reform proposal did away with content categories and created, in their place, categories centered on abilities development and a basic 
goal statement: that general education should help students learn to think critically, reason logically, and communicate effectively. 
The proposal also removed all courses from the program and created a contractually recognized oversight body to evaluate courses 
submitted to the new program. By a narrow vote of the full university faculty, this first reform proposal was defeated. 

After listening closely to various voices in the debate, a reconfigured group led by Professors Gardner and Dome, submitted a second 
proposal the next semester. Though this proposal maintained the content categories (albeit in a revised form), it kept several large* 
scale reform elements; for example, it articulated not only an overall program goal, but also revised intersecting category goals and 
created descriptive criteria forcategories; it created an oversight body with significant powers; it removed allcourses from the program 
and required courses to be submitted to the new program; it allowed courses accepted into the program a five-year term and set up 
ongoing assessment mechanisms. 

In a semester-long process, which included informational mailers, more than a hundred e-mail inquiries about the proposal, two lively 
campus-wide forums, presentations at formal curriculum committee meetings, and several meetings with individual faculty members 
and administrators, the proposal was revised and then unanimously approved by the full university faculty and administration in 
March 2000. The faculty union and the administration then negotiated a memorandum ofunderstanding(an addendum to the faculty 
contract) that clarified procedural matters relevant to general education, especially the formation of the general education committee 
(GEC). This new GEC began accepting faculty proposals for courses, and SVSU first offered courses under the new GE program in fall 2001. 



The General Education Committee (GEC) 

Like other standing committees at SVSU, the GEC is composed of six elected faculty members (one from each of the curricular units), two 
administrativeappointments(oneofwhomchairsthecommittee), and onestudent representative. It ischarged with three main functions: 




1. To receive and evaluate GE course proposals submitted by the academic departments for possible inclusion in the GE 
program. The evaluation process includes providing written feedback to the academic departments. 
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2. To receive and evaluate GE course assessment plans and assessment reports submitted by the departments. The GEC 
provides written feedback on both types of documents to the academic departments. 

3. To report to the faculty and to the vice president for academic affairs and to advise on the planning of GE course schedules. 

The New General Education Program 

The new program currently has fifty-eight courses organized in ten content categories (literature, arts, numerical understanding, 
natural sciences, historical and philosophical understanding, social sciences, social institutions, international systems, oral 
communication, and written communication). All courses currently in the program went through the rigorous course approval process 
governed by the GEC and received five-year terms in the program. When courses are revised and/or resubmitted during or after this 
term, submitters are required to make use of ongoing assessment information. Course proposals were required to speak to the 
program's main goal: “to develop students' capacity to think critically, reason logically, and communicate effectively in a wide range of 
disciplines beyond their major field.” Courses were also required to speak to a learning objective specific to the course's category (e.g., 
all courses in the literature category ask students to learn “to read major literary works critically with appreciation and understanding”). 

In addition to a specific student learning objective, each category is governed by a set of descriptive criteria. For example, literature 
courses must 

o Offer practice with strategies of close reading and analysis of texts 

o Consider a range ofways of respondingto texts (which may include discussion of aesthetics, values, and ethics as they relate 
to texts) 

o Consider texts' historical context 

o Introduce students to a range of genres (e.g., fiction, poetry, drama) or a single genre examined over an extended period 
oftime 

o Consider literature from multiple cultures 
o Meet or exceed all of the criteria for communication-intensive courses 

Beyond the category and distribution requirements, the program calls for students to take three courses (one in literature and two 
chosen from the remaining categories other than oral and written communication) that are designated “communication-intensive” 
courses; such courses are required to meet additional criteria tied to written and oral communication. Students are also required to 

completetheircoursesinnumericalunderstanding,oralcommunication,andwrittencommunicationpriortocompletingseventy-five 

total credit hours. 

Assessing the New Program 

SVSU is assessing the new program in three ways. 

0 DepartmentaUy-generated assessment. Departments that have courses in the program are required to submit assessment 

plans to theGE^ which then provides feedbackand counseling. Assessment plans specify the ways in which student learning 
(in terms ofboth the program's main goal and the course's category-specific objective)will be measured, ideally through both 
quantitative and qualitative means. After plans are approved, departments are asked to submit annual reports on their 

assessmentwork; as with assessment plans, theGEC offers feedbackandcounselingonthesedocuments.TheGECmaintains 

plans and reports for faculty across campus to use as models. 

During the first year of GE assessment, most departments were able to comply with the assessment policy. The remaining 
departments experienced difficulties in two areas: the need to fully and overtly integrate the goals of the GE program into the 

course planningprocess, and the need todevelopmeasuresthatspecificallyspoketoprogramgoals (in additionto discipline- 

specific or course-specific goals). 

0 Standardized testing and surveying. In fall 2002 the SVSU Assessment Office, working with the GEC and the academic 
departments, fully implemented two standardized instruments that were initially piloted in fall 2001: three subtests of the 
Collegiate Assessment of Academic Proficiency (critical thinking, writing skills, and writing: essay) and the Cooperative 
Institutional Research Program Freshman Survey. Initial data are now being analyzed and compared to information in 
departmentally generated assessment work. 

0 Assessment of processes. Additionally, the GEC is continually assessing the course submission and assessment oversight 
processes. A key early piece in this process was the institution of a series of faculty focus groups held this past summer. 
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Closing the Loop 

The GEC is now beginning to issue annual reports to the university community that summarize work in the program. We expect that, 
like all living programs, our general education program will need to continuously improve, and the GEC sees one of its central goals 
as creating and widely sharing a paper trail of assessment information to aid in future revisions. 



Eric Gardner is Associate Professor of English and Self-Study Co-Editor at Saginaw Valley State University in University Center, 
Michigan, 

Diane Boehm is Director of Instructional Support Programs at Saginaw Valley State University in University Center, Michigan, 

Clifford Dome is Assistant Dean oftheCollegeofArtsand Behavioral SciencesatSagina'wValley State University in UniversityCenter, 
Michigan, 

Mary Hedberg is Assistant Vice President of Academic Affairs and Self Study Coordinator at Saginaw Valley State University in 
University Center, Michigan. 
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Assessing the Achievement of 
General Education Objectives: 
Four Years of Assessment 



Irene Kovala, Trudy Bers, Gary Mines, and Suzanne Stock 



The Setting 

Oakton Community College is a public community college located in the north shore suburbs of Chicago. The college's district is 
affluent, with nationally known public schools and a strong commitment to education. The combination of the high socioeconomic 
status of residents and expectations that high school graduates will go to college results in a college-going rate ranging from 80 to 95 
percent among recent high school graduates, including even those who do not meet traditional academic standards for college 
success. 

The student population of Oakton Community College includes a large percentage of foreign-born students whose native language is 
not English-upwards of 20 percent-and a large percentage of students who already hold a bachelor's degree or above-about 20 
to 25 percent. Eighty percent of students are part-time; students' average age is thirty, and 45 percent are above the age of twenty- 
four. Each fall 30 percent of students are new to the institution, including reverse transfers who previously attended other colleges. 
In the spring the percentage of new students drops to about 12 percent. 

Two-thirds of students are in baccalaureate-transfer programs, though this figure is somewhat misleading, since many bachelor's 
degree students are categorized in transfer programs even though they are taking courses primarily to learn job-related skills or for 
personal enrichment. As is the case at most community colleges, only a small percentage of students complete associate degrees or 
certificates (about 5 percent of the total student population). Most students who transfer do so before earning a degree. 



Project Description 

In spring2oo2 a general education subcommittee of the Oakton Student Academic AssessmentTeam (SAAT) conducted the fourth year 
of its project to assess general education learning outcomes. Students read or watched a prompt, then answered questions about the 
prompts. Prompts and questions were designed to assess students' achievements on four general education objectives: 



o Construct hypotheses (two different prompts were used to assess achievement of this objective) 
o Analyze and interpret data and information 

o Recognize skills necessary to build and maintain effective human relationships 
o Effectively communicate ideas in writing 

Prompts for the first two objectives were written. The last two objectives were assessed using the same prompt, which was a video clip 
from the film Ordinary People. Trained faculty used scoring rubrics to assess students' answers. Rubrics were based on a three-level 
scoring system, with the top two levels indicating that a student met standards for general education objectives addressed on that 

prompt.Twodifferentscoring rubricswereusedtoscorethestudents'answerstothe film clip questions, one dealingwiththeobjective 

of recognizing skills necessary to build and maintain effective human relationships and the other dealing with the objective of 
effectively communicating ideas in writing. 




Students in eighty-eight classes with large numbers of students who had earned thirty or more credits at Oakton participated in the 
2002 generaleducation assessment. Atotalofi,i65 students responded to oneor more ofthegeneraleducation assessment prompts, 
and a total of 1,664 responses were scored. Recall that responses for one prompt were scored twice, and a handful of students may 
have completed more than one prompt. 
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We obtained data forthe project from both the students* scores on their responses and the college's Student information management 
system. The dependent variable is the student's achievement on the prompt. We also identified a number of demographic and 
academic variables to use as controls: age, gender, cumulative grade point average, cumulative college-level credits at Oakton, 
English composition proficiency, mathematics proficiency, and number of general education areas in which students had taken 
courses either at Oakton or elsewhere (this was based on students' self-reports).^ 



Results 

Researchers examined the association between a number of student demographic and academic characteristics and students' 
achievement levels on the prompts. 

0 Overall, 67 percent of responses received passing scores; that is, students met standards on general education objectives. Of 
all responses, 345 (21 percent) were scored as high passes, 767 (46 percent) were scored as lowpasses, and 552 (33 percent)were 
scored as no passes. The number of responses scored for each prompt and percent of students who passed is presented in the 
table. 





Objective: 

Construct Hypotheses 
Prompt: 

AIDS and drug use 

N=294 


Objective: 

Construct Hypotheses 

Prompt: 

Fetal heart rate 
N=26 i 


Objective: 

Analyze and Interpret 
Data/Info 

Prompt: 

Circle graph- 
age and AIDS 
N=386 


Objective: 
Human Relations 
Prompt: 
film clip 
N=3S6 


Objective: 

Communicate Clearly 
in Writing 
Prompt: 
film clip 
N=36? 




N 


Pet 


N 


Pet 


N 


Pet. 


N 


Pet 


N 


Pet. 


High Pass 


41 


14 


53 


20 


126 


33 


38 11 


87 


24 


Low Pass 


130 


44 


116 


44 


168 


42 


182 


51 


171 


47 


No Pass 


123 


42 


92 


35 


92 


24 


136 


38 


109 


30 



0 Fifty-three percent of students whoseworkwas scored for both recognizing skills necessaryto build and maintain effective human 

relationships and effectively communicating ideas in writing— the same prompt was used for both objectives— received passing 
scores on both rubrics applied totheprompt.Scoringthesameresponsewith two rubrics enables ustoexplorewhetherthesame 
student has achieved standards on two general education objectives. 

0 Statistically significant associations' between independent variables and achievements on general education objectives were 
as follows. 

o Genderwas associated with achievements on the objectives of human relations and effective writing (film clip prompt). 
A higher percentage of females than males received passing scores when answers were scored on both the human 
relations rubric (68 percent of females passed and 53 percent of males passed) and the effective writing rubric (78 
percent of females passed and 60 percent of males passed). 

o Age was associated with students’ responses to the AIDS and drug use prompt to assess achievement of the general 
education objective of constructing hypotheses; 62 percent of students under the age of twenty-five and 49 percent of 
students age twenty-five and older who responded to this prompt received passing scores. 

o Cumulative credits earned at Oakton were associated with only one prompt, the fetal heart rate prompt to assess 
achievement of the general education objective of constructing hypotheses. Students who had earned either fewerthan 
thirty credits or forty-five or more credits had the highest pass rates (68 percent and 70 percent respectively), whereas 
only 43 percent of students who had earned thirty to forty-four credits received passing scores. 

o Cumulative grade point average at Oakton was associated with achievements for the two general education objectives 
of effective writing and human relations. Students who had higher grade point averages were more likely to achieve 
standards. 

o Students' competence in mathematics was not associated with achievement on any of the objectives. 

o Students' competence in English composition was associated with achievement on all objectives except the one for 
human relations. Students with college-level competency and those for whom there was no information about 
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competency received a substantially higher percent of passing scores than students with remedial-level competency 
(relatively few students were categorized at the remedial-level). 

o The number of general education areas in which a student reported taking courses was associated with achievement on 

two objectives: constructing hypotheses as measured by the prompt about AIDS and drug use, and analyzing and 
interpreting data and information. Overall, most students reported having taken courses in three or more general 
education areas, and generally they had higher percentages of passing scores than students who reported taking 
courses in fewer general education areas. 

0 The percentage of students who achieved passing scores on the general education assessment varied in each of the four years 
of this general education assessment project (1999-2002). In 1999, 59 percent passed; in 2000, 77 percent passed; in 2001, 64 
percent passed; and in 2002, 67 percent passed. In 2000, one prompt was answered in a multiple-choice format, and 95 percent 
of students achieved passingscores, skewing results forthatyear. Differences in the prompts, scoring rubrics, general education 
objectives, instructions to students, and reader procedures across the years make it inappropriate to compare achievements 
from one year to the next. 



Factors Fostering the Success of the Project 

Five key factors were instrumental in fostering success of the assessment project: (1) high-level faculty and administrative leadership 
of the Student Academic Assessment Team and faculty involvement, (2) moving the assessment of general education outcomes to a 
college rather than departmental level, (3) initiatingthe project as a pilot, (4) refusal to attribute blame, praise, or excuses for student 
performances, and (5) a four-year record of implementing the assessment— that is, beginning to institutionalize it. 

First, high-level faculty and administrative leadership of the Student Academic Assessment Team and the involvement of faculty in 
developing and scoring the prompts demonstrated both institutional commitment and faculty buy-in to the project. The team has 
always been co-chaired by the vice president for academic affairs; the first co-chair was the faculty senate president, and the current 
co-chair is another faculty member (the current faculty senate president is a member of the team). 

Second, by having the Student Academic Assessment Team take the lead role in assessing general education outcomes, individual 
departments teaching general education courses are able to concentrate their assessment efforts at the classroom, course, and 
program levels. This approach and the fact that students in a variety of classes, not just general education courses, participated 
emphasize that general education is a college-wide, not a departmental, responsibility. 

Third, the college initiated and implemented this assessment project as a pilot. Doing so fostered senses of experimentation and 
relief. Faculty participants recognized that because this was a pilot, the process could be modified to accommodate unforeseen 
difficulties or experiences. Conceptualizing the project as a pilot implicitly gave the institution permission to make mistakes as part 
if its own learning experience. Though not planned, the SAAT members and the prompt readers formed mini-learning communities 
in which the topic of their learning was general education assessment. By the third and fourth year the term pilot ms dropped. 

Fourth, in evaluating results, there has been no attribution of praise, blame, or excuses for student performances. Faculty members 
seem to recognize that results may be due to a combination of factors: weaknesses in the assessment project itself, students’ not 
taking their work on the assessment prompt seriously; lack of writing and communication competency; and other reasons may have 
allcontributed. This recognition has begun tospark good discussionsamongfaculty about assessment as a processand about student 
learning outcomes and competencies. 

Finally, the project has been conducted for four years, beginning in 1999. Though the nature of prompts and the details of 
implementation varied, the general approach remained the same. While it is difficult to know exactly when a “pilot” becomes 
“ongoing,” and when an “ongoing” project becomes “institutionalized,” the college continues to be committed to assessing general 
education learning outcomes across the institution under the umbrella of the Student Academic Assessment Team. 



Next Steps 

Two important next steps relate to faculty discussion about general education assessment results and a revised approach to be 
implemented in spring 2003. 

0 Faculty discussions. Early in the spring 2003 semester, faculty will be asked to discuss assessment of courses, programs, 
and general education outcomes at departmental or division meetings. One purpose in convening these discussions is for 
faculty, more than a third of whom joined the college within the past three years, to become more familiar with assessment 
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overall. A second purpose is to engage faculty in considering how they might incorporate teaching general education 
objectives in courses across the curriculum, not just in traditional general education disciplines. 

0 Revised approach. The SAAT subcommittee is planning to broaden its approach to assessing general education outcomes 
in spring 2003. The specific objectives to be assessed in 2003 are 

o Communicate...effectively in...speech [presentations! 

o Work and communicate effectively with people from a variety of backgrounds, individually and in teams 

The subcommittee will be identifying classes in which assignments involve presentations or working in groups (teams). Faculty 
teaching the classes will be asked to volunteer to have students’ presentations or work in teams assessed by trained observers using 
a scoring rubric. Their assessments will be based on regular classroom assignments but will not be part of students’ grades. This 
revised approach should enable the college to begin to assess students’ achievement of general education outcomes that do not lend 
themselves to paper-and-pencil responses or tests. 



Notes 

‘ Most variables need no further explanation; however, two warrant elaboration. English competency was constructed as a proxy to 
depict students’ proficiency in English composition based on courses and placement tests at the institution. Transcript and 
placement test records were examined, and students were placed into mutually exclusive categories in descending order of 
demonstrated proficiency. That is, all students who had received grades of A, B, or C in a college-level composition course were 
considered to be in the highest category of composition proficiency; those who had enrolled or placed into a college-level 
composition course were in the next highest level, and so on. Students for whom no data were available were arbitrarily assigned 
to the lowest category. The same approach was used to classify students according to their competency in mathematics, with 
intermediate algebra considered separately from college algebra and other college-level courses because it is not typically accepted 
for transfer and is a prerequisite for other college-level courses. 

'“Statistical significance” simply means that the association between the independent and dependent variable is stronger than we 
would expect from chance alone. Statistical significance is often attained when sample sizes are large, even when there is no 
practical or substantive significance. We took some latitude in identifying statistical significance because in some instances the 
number of cases in a cell was quite small, indicating that the chi-square may not be a valid test. 



Additional information 

Additional information about the Oakton general education assessment is available in these resources. 

Bers, Trudy H. 2000. Assessing the achievement of general education objectives: Second year of a college-wide approach and lessons 
learned, journal of Applied Research in the Community College 7(2). 

Bers, Trudy H. 2000. Assessing the achievement of general education objectives: A college-wide approach. Journal of General 
Education. 49(3). 

Bers, Trudy H. 2002. Assessingtheachievementofgeneraleducation objectives: Fouryearsofassessment. Unpublished Report. Office 
of Research, Oakton Community College, September. 

Bers, Trudy H., B. D. Davis, and M. L. Mittler. 2001. Assessing the achievement of general education objectives in the community 
college: A project across the disciplines. Assessment Update, May/june. 

Also see <http://www.oakton.edu/resource/oir/resmenu.html>. 
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Effective General Education Assessment 
at Large Public Institutions 

julie Wallin and Brenda Masters 



Introduction 

General education assessment presents unique challenges compared to assessment within a degree program, particularly at large 
public institutions. Although the ideal is to approach general education assessment in the same manner as program outcomes 
assessment, the nature of general education often presents obstacles that make its assessment vastly different from assessment in 
the major. For example, general education practices at larger institutions may include these obstacles: 

o Students select a few courses from hundreds that are designated as part of the general education curriculum, 
o Students often transfer general education credit hours from other institutions, 
o Some general education courses are large and may be taught by non-faculty instructors, 
o Few faculty members claim ownership of the general education curriculum as a whole. 

o The process for evaluating or designating general education courses does not incorporate information from student learning 
assessment. 

o Logisticalconstraints preventinstitutionsfrom requiringstudents to participatein assessments thatoccuroutsideofclasses. 
o Students rebuff participation in assessments not related to grades. 

A great deal can be learned from examining how peer institutions approach general education assessment, facilitate faculty 
leadership of general education assessment, and use general education assessment results to effect change. We present these 
aspects of general education assessment at Oklahoma State University (OSU), a large, public, land-grant institution. 



Development of the General Education Assessment Model 

As a result of a state regents* mandate and regional accreditation requirements, OSU has used several general education assessment 
tools over the past decade, including standardized tests, student surveys, and student portfolios. However, none of these has evolved 
into a sustainable process for long-term general education assessment because of logistical problems associated with methods 
themselves (including student participation) and difficulty in effectively using the results for evaluating or developing the general 
education program as a whole. 

In 2000 a faculty task group was appointed to develop an effective and sustainable general education assessment program for OSU. 
The group members represented a range of disciplines and were paid summer stipends. Following the assessment standard of 
articulating desired student outcomes first (Taylor & Marienau, 1997), the group started by revising OSU*s "Criteria and Goals for 

General Education Courses'* documentand identifying assessable outcomes forthe general education program. Most importantly, the 

group developed guidelines for effective and sustainable general education assessment that included these principles: 

o The process must not be aimed at individual faculty members or departments, 
o The process should be led by faculty members, and faculty participation should be voluntary, 
o The process should use student work already produced in courses. 

o The process should assess all undergraduates, including transfer students, because general education outcomes describe 
qualities expected for all students. 
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This last principle is particularly important because it articulates the view that students’ general education skills (writing, problem 
solving, etc.) are not exclusively developed in the general education curriculum and should not be assessed solely within those 
courses. 

After studying general education assessment methods from other institutions, the faculty group developed a model for general 
education assessment that included assessments that met these guidelines. The model included institutional portfolios for 
evaluatingstudent achievement ofgeneral education outcomes, surveys that provide indirect information about student achievement 
of those outcomes, and an evaluation the general education curriculum via an online course content database. 





Institutional Portfolios: Evaluatingstudent Achievement of General Education Learner Goals 

OSU currently evaluates students’ written communication and math problem-solving skills via institutional portfolios, and portfolios 
are being developed to evaluate science skills and complex problem-solving skills. The institutional portfolios are patterned after 
those used at )ohnson County Community College (Seybert & O’Hara, 1997). A separate institutional portfolio is (or will be) developed 
to target each general education outcome (e.g., written communication skills), and each portfolio includes a random selection of 
student work demonstrating that outcome in a range of courses (general education and major courses). The sampling is loosely 
stratified so that the portfolio includes assignments from a variety of disciplines and levels (freshman- through senior-level work). 
Faculty members evaluate the students’ work in each portfolio using a scoring matrix (rubric) that describes the general education 
competency being assessed. By rotating the faculty reviewers so that continuing members teach the portfolio scoring techniques to 
new members, the review process is standardized from year to year. This is essential for reasonably combining the institutional 
portfolio data over years. 

The two years of writing assessment data have produced an overall pattern of student writing skills where, on a 5-point scoring scale, 
66 percent of all students demonstrate writing skills of 3 or higher and 21 percent of all students demonstrate writing skills of 4 or 
higher. The data show patterns of improvement in writing skills from freshmen to seniors and show that writing skills of students who 
start at OSU as first-time freshmen are somewhat higher than writing skills of transfer students. Most importantly, these preliminary 
data demonstrate how institutional portfolio results may be analyzed to evaluate writing skills or other general education outcomes 
in the student body. Institutional portfolios seem to provide reliable measures of student learning, especially if the portfolios are 
evaluated by faculty who are trained to use a rubric to standardize their scoring of student work. 

Although longitudinal portfolios that track the work of /nd/V/doo/ students are widely used as local assessment measures (El-Khawas, 
1995), this type of institutional portfolio is less common and provides more of a snapshot of skills demonstrated in the student body. 
Institutional portfolios do not demonstrate individual student development, but they are logisticallyeasierto produce; they represent 
the entire student body (including transfer students); and they provide many of the same advantages as individual portfolios-they 
directly assess learning and are evaluated based on criteria established by faculty (Palomba & Banta, 1999). Institutional portfolios 

at OSU arestartingto provide useful information for program improvement. Forexample, the faculty committeethatsets the standards 

for general education courses is considering modifying the criteria for general education courses in response to the institutional 
portfolio findings, and several faculty involved in the review process have made department- or course-level changes as a result of this 
assessment process. 
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Surveys: Indirect Assessments of General Education 

OSU’s general education assessment model also includes indirect evidence of student achievement of general education outcomes 
via surveys. OSU participates bi-annually in the NationaiSurvey ofStudent Engagement (NSSE, administered by the Indiana University 
Center for Postsecondary Research Pianning) and Coiiege Student Survey (administered by the University of California Los Angeles 
Higher Education Research Institute), and the university coordinates annual alumni surveys that target recent graduates. Survey 
resuits complement the information coilected from institutionai portfolios. For example, NSSE results indicate that OSU students do 
not complete as many writing assignments asstudents at peer institutions, and this hascontributed tothe dialogue regardingfindings 
from the writing skills assessment portfolio. 



General Education Course Database: Evaluating the Curriculum 

Evaiuatingthe curriculum itself complements assessments of student learning. The Web-based General Education Course Database 
was developed to evaluate course content and determine how each generai ed ucation course is aligned with specific expected general 
education learning outcomes. The course content database replaces a long-standing paper-based course review process and has the 
advantage of utilizing data mining techniques for examining the general education curricuium in a variety of ways. A faculty advisory 
councii reviews all general education courses every three to five years. The reviews are based on course information provided by 
instructors via online database forms. Specificaiiy, instructors identify how their course incorporates one or more of the expected 
outcomes for general education and howstudentachievementofthatoutcomeisevaiuated within the course. Instructors aiso identify 
course activities reiated to generai education goals (e.g., how often do students write papers, give orai presentations, discuss how 
the course content reiates to contemporary issues). 

The database now contains information from about 30 percent of all general education courses in the curriculum, and the remaining 
courses wiii be added in the next academicyear. When compieted, the database wiil be used to evaiuatehowand where thecurricuium 
addresses each generai education iearner goai. Combined with transcript analysis, the database will also show the frequency with 
which students enroll in courses targeting specific general education outcomes and how frequently students are exposed to particuiar 
general education concepts. Although the database does not provide a measure of student learning, it has become a criticai tooi for 
examining the curricuium reiative to specific general education outcomes. 



Why the Model Works 

Now in its third year of implementation, the OSU assessment model is beginning to yield useful information and gain faculty support 
and momentum. Although its sustainability is still unproven, the aspects of the model that are working are evident. They include the 
following. 

0 Facultyleadership.The model has been developed entirely byfacultyrepresentingavarietyofdisciplines.Thechief academic 
officer has supported this group; faculty receive summer stipends for their work on general education assessment; and the 
group has considerable logistical support for its activities from OSU’s central assessment office. The group has established 
membership rotation to ensure the continuation of faculty support, and the portfolio reviews include increasing numbers of 
faculty members each year. 

0 A holistic approach with multiple measures. The model presents a holistic approach to evaluating general education and 
carefully avoids focus (and potential blame) on individual disciplines or faculty groups. Student learning assessments 
(institutional portfolios and surveys) examine student progress toward attaining general education outcomes university- 
wide, and the course evaluation process (via the course content database) looks at how frequently general education courses 
include opportunities for students to develop particular general education skills. If or when these assessments converge on 
similar findings (e.g., the need to focus attention on student written communication skills), faculty leaders are empowered 
to identify and implement steps toward program improvement. 

0 Using student work already produced in the curriculum. Institutional portfolios are now a cornerstone of OSU’s general 
education assessment program, and as Jeffrey Seybert has aptly stated about institutional portfolios at Johnson County 
Community College, the portfolios “obviate the need for motivating students to participate in assessment.” Institutional 
portfolios are transparent to students, protect student privacy, and accurately represent the student body in assessment 
results. 

0 Communicating results. The faculty group has established processes for communicating results of its work through an 
annual newsletter for faculty and annual presentations to administrative and faculty groups. 
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Obstacles to General Education Assessment and Alternative Solutions 

At this point in the presentation, session attendees at each table will be asked to identify and discuss, based on experiences on their 
own campuses, other approaches to and obstacles to general education assessment. The session leaders will circulate among groups 
and record alternative methods and obstacles. The remainder of the session will be an audience-participation discussion of different 
general education assessment tools, factors that contribute to success or failure of general education assessment initiatives, and 
other examples of how institutions have overcome the common obstacles to effective general education assessment. 



o The general education assessment model at Oklahoma State University includes a variety of methods. Institutional portfolios 
provide direct evidence of student learning; surveys indirectly assess students’ general education experiences: and the 
curriculum can be evaluated via a course content database. The use of multiple assessment methods provides a more 
accurate gauge of student learning outcomes than any method individually. 

o This general education assessment approach satisfies basic principles established by faculty for sustainable assessment. 
The process is led by faculty; faculty participation is voluntary; and the primary assessment tool uses student work already 
produced in courses (including general education courses and major courses). The process holistically evaluates general 
education and encompasses all undergraduates, including transfer students. 

o Information generated from assessment should be thoroughly shared with various campus constituents where it can be used 
to effect positive change in the learning environment. 
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Update, 9(4): 5-7. 

Taylor, K., and C. Marienau. 1997. Constructive-development theory as a framework for assessment in higher education. Assessment 
and Evaluation in Higher Education 22(2). 



lulie Wallin is the Director of University Assessment at Oklahoma State University in Stillwater. 

Brenda Masters is Associate ProfessorofStatistics and Director University Re-Accreditation at Oklahoma State University in Stillwater. 



Overview 



References 



ERIC 




Chapter 4. General Education: Assessing Outcomes, Reforming Programs/ 131 



A$$essing the Ineffable 
Outcpn^es of General Education 
Through Electronic Student Portfolios 

Sharon J. Hamilton 



This session addresses the confluence of three facets of higher education: general education, assessment, and electronic student 
portfolios. It responds to the challenge set forth by Richard Shavelson and Leta Huang in the January/February 2003 issue of Change, 
wherein they assert “the absence of a coherent conceptual framework that would align assessments with the valued outcomes of 
highereducation”(p.ii).Thissessionwillpresentoneexampleofacoherentconceptualframeworkthat attempts to align assessment 
and valued learning outcomes that transcend’domain-specific knowledge. 



A Brief Background 

Individual faculty are variously experienced and adept at assessing domain-specific knowledge and abilities, but much less 
experienced and adept at assessing ways of knowing across a range of domains. Departments, academic programs, and institutions 
are increasingly interested in assessing knowledge and abilities across a range of domains. Some institutions use standardized tests, 
such astheCAAP(ACrs Collegiate Assessment ofAcademic Proficiency)theETSTasks in CriticalThinking,ortheETS Academic Profile. 
Increasingly, however, institutions are turning to electronic student portfolios to assess student learning both within and across 
domains. Alverno College and Rose-Hulman Institute of Technology have developed quite different yet equally impressive electronic 
student portfolios to show both improvement and achievement in student learning. 

Recently, several national consortia have been instituted to share the enormous cost of developing electronic student portfolios. 
Additionally, AAHE, NLII, and other organizations have been establishing nationwide networks of faculty, staff, and administrators 
involved with the development of electronic student portfolios. These networks and consortia address issues of cost, security, 
transportability, and efficacy, and share responsibilities for the development of different functions within the portfolios being 
developed. 

Electronic student portfolios are as variable as the institutions involved in their development. While most have several functionalities, 
each one generally focuses on a primary function, whether it be advising, career search, assessment of learning, or demonstration of 
proficiencies. 

While acknowledging the broader context of portfolio development across the country, this session will focus primarily on the 
development of the lUPUl portfolio, which attempts not only to demonstrate and assess improvement and achievement in student 
learning but also to catalyze deeper and more coherent learning within the major and across subject domains. 



The Inef^bles of Undergraduate Education 




Faculty-based assessment of individual students focuses almost exclusively on cognitive abilities, and many will argue that this is 
exactly as it should be. At the same time, most faculty will also agree that education is about much more than what is generally 
assessed on final examinations. If we were to gather groups of faculty from around the nation, and ask each group to identify what 
undergraduate education hopes to achieve, most would come up with fairly similar lists: students emerging from higher education 
with baccalaureate degrees should have knowledge and skills that go beyond the cognitive and that include personal, social, and civic 
attributes. Expectations for our graduates would include enhancing values and ethics, critical thinking, integrating and applying 
knowledge, responsible citizenship, appreciating diversity, or aesthetic discernment. Yet we rarely assess improvement and 
achievement in these areas. We rarely even try to agree on what we mean by them, which is why they are often referred to as “the 
ineffables of education.” But without these Ineffables, of what value is cognitive proficiency. As Theodore Hamerow wrote in 
Reflections on History and Historians (1987): 
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Himmler... knewtheclassics well, but they did notteach him much kindness. Goebbels had studied literature a ndhistory...without 
learning generosity or compassion. What the humanities bringto the student is what the student brings to the humanities. If he 
likes art or music, they may make him more appreciative;...if he is interested in literature, they might make him more 
knowledgeable...But they cannot teach wisdom or virtue; they cannot...mold human character. 

Some of us may respond that that is exactly right: we are not in the business of teaching virtue or molding human character. Others 
amongusmay wonder, though, what education isallaboutif it does not shapecharacteror lead toward virtue and wisdom. It’s a thorny 
issue, and one not likely to lead to agreement, particularly within the confines of one presentation. But, for many of us, it is among 
these ineffables of education that the value and coherence of education reside. 

This presentation will show how the lUPUl electronic student portfolio presents a conceptual framework that integrates domain- 
specific cognitive knowledge with the cross-disciplinary ineffables of learning to demonstrate and assess improvement and 
achievement in both. 



The Ineffables of Education at lUPUl 

In 1998, after almost ten years of dialogue involving several hundred faculty members, the lUPUl Faculty Council approved six 

principles of undergraduate learning (PULs): 

1. Core communication and quantitative skills. The foundational areas of writing, reading, speaking, listening, quantitative 
analysis, and use of information technology-the core skills for lUPUl students-are demonstrated, respectively, by the 
ability to 

a. Express ideas, opinions, beliefs, and facts to others effectively in a variety of written formats (i.e., basic writing 
composition, general written communication, and professional or research writing) 

b. Comprehend, interpret, and analyze written text in reading 

c. Communicate effectively (speak and listen) one-on-one and in small and large group settings, as well as identify factors 
that facilitate and impede communication 

d. Perform quantitative functions and analyses 

e. Use information technology for academic, personal, and professional needs 

These foundational skills are introduced in specific courses and developed and extended throughout the disciplines. 

2. Critical thinking is a sophisticated cognitive process that involves the careful examination of ideas and information from 
multiple perspectives in order to clarify and improve understanding and to develop ideas that are unique, useful, and worthy of 
further elaboration. Critical thinking is demonstrated by 

a. Solving challenging problems 

b. Analyzing complex Issues and making Informed decisions 

c. Synthesizing Information to arrive at reasoned conclusions 

d. Evaluating the logic, relevance, and validity of data 

e. Using knowledge and understanding to raise and explore new questions 

3. Intellectual depth, breadth, and adaptiveness is the ability to examine, organize, and apply disciplinary ways of knowing to 
specific issues. Intellectual depth is demonstrated by substantial knowledge in one area, usually the major, but, where 
applicable, in a minor orotherconcentratlonofstudy. Intellectual breadth is demonstrated bytheabilitytocompare and contrast 
approaches to knowledge in different disciplines and by the ability to define what counts as evidence in each discipline. 
Adaptiveness is demonstrated by modifying one’s approach to a problem or question based on the requirements of a particular 
situation. 

4. Integration and application of knowledge. Integration of knowledge is demonstrated when students articulate and apply 
concepts or constructs from two or more disciplinary areas to personal, academic, professional, or community activities. 
Application of knowledge occurs when students participate in experiences that enable them to link their knowledge to their own 
Intellectual development, to their professional goals, and to the goals of society. 
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5. Understanding society and culture is the ability to place one’s own cultural traditions in a broader human context. This ability 
is demonstrated by writing, actions, and speech that indicate knowledge of the range of diversity in traditions, history, and 
values. 

6. Valuesandethics.Anundergraduateeducationfbstersthedevelopmentofasenseof aesthetics, values, and ethical standards. 

The enactment of values and ethics occurs when students make informed and ethical decisions in their personal, academic, and 
professional endeavors. 

In 2001, the faculty committee guiding the development of the electronic student portfolio decided that one way to show improvement 
and achievement in the PULs was to determine what all students should know and be able to do in relation to each PUL at three levels: 
introductory (by the end of twenty-six credit hours), intermediate (by the end of fifty-six credit hours; the equivalent of an associates 
degree), and senior (before graduation). We established a multidisciplinary committee for each of the PULs to develop campus 
consensuson the first two levels regardless of major orprofessional program, leaving each departmentor academic program to define 
competence at the senior level. Recognizing that growth in the ineffables of education also occurs in co-curricular, extra-curricular, 
and other activities, we added a fourth level: the experiential level, students can select for their portfolios examples of growth in 
learning from out-of-classroom experiences (that may well have been influenced by in-class experiences). 

Coming to multidisciplinary consensus on learning outcomes related to ineffable goals such as critical thinking, values and ethics, 
integrating and applying knowledge, and valuing diversity has been no easy task. Even so, almost a hundred faculty members have 
been engaged in this process forthe past six months. Once their reports are in, they will be distributed to all academic and professional 
schools for further refinement and then sent through the faculty governance process. 

Assessment of improvement and achievement in the PULs will be carried out according to the consensually developed learning 
outcomes articulated by the multidisciplinary campus committees. The learning matrix of the electronic student portfolio will provide 
the mechanism for this demonstration and assessment of improvement and achievement. 



The lUPUl Learning Matrix 

The learning matrix is at the core of the lUPUl electronic student portfolio. Students select and upload completed and graded 
assignments from theircoursesthat they thinkdemonstrate proficiency inone (or more) facets ofone (or more)ofthe PULs. In so doing, 
they come to see how these PULs transcend any one course or major while at the same time realizing that they are deeply embedded 
in many courses and academic programs. 
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There are several significant features to this matrix. 

o It provides an opportunityto demonstrate both improvement and achievement in the PULsusingcompleted assignments from 

students’ regular courses. 

o It tracks improvement and achievement in all the PULs throughout the student’s academic career on one page. 

o It acknowledges that learning occurs outside as well as inside the classroom by allowingfor co-curricular and extra-curricular 

examples in the experiential column of the matrix. 

o The reflective component enhances learning by serving as a catalyst for metacognitive thinking about the documents the 
student chooses to demonstrate proficiency. 

o It will be integrated with the registrar’s office, so that students will receive automated replies whenever they upload a 
document and automated congratulations whenever they complete a cell in the matrix. 

o Each completed cell will turn a different colorfrom the uncompleted cells, so that students can see exactly where they are in 
relation to the PULs. 

o Completed cells will be e-mailed automatically to one of the evaluators (most likely a member of the lUPUl Senior Academy 
of retired faculty members who want to stay involved with the campus), who will respond directly to the students’ reflective 
writing. This interaction beyond the bounds of the classroom should further enhance learning and student motivation. 

o For department-wide, program-wide, or campus-wide assessment, completed cells may be aggregated and assessed in order 

to track improvement and achievement in student learning beyond the level of the individual student. 

While the lUPUl electronic student portfolio is still in its developmental stages, we are excited about its potential to demonstrate and 
assess improvement and achievement in student learning, as well as to enhance deeper learning and provide coherence to the 
undergraduate curriculum. In a university with more than twenty academic and professional schools, this is quite a challenge. We look 
forward to hearing the responses and ideas of participants in this session, to learning from their questions and concerns, and to 
working together to develop meaningful ways to assess what we value in higher education. 
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Assessing Critical Thinking 
In the General Education Program 

Becky J. Chadwick and Kim D. Schopmeyer 



The Importance of Critical Thinking in General Education 

In the last two decades, critical thinking skills have increasingly become identified as key components of general education in 
elementary, secondary, and higher education. It is no longer considered sufficient for students to simply acquire a body of facts and 
information. Rather, educators and employers recognize that education should lead to broader abilities to analyze and evaluate 
information and to develop approaches to solving problems. Moreover, critical thinking skills are often cited as essential to effective 
citizenship, allowing members of society to understand key social issues and problems and make informed decisions. 

As a result, critical thinking is frequently embodied in the mission statements of colleges and in general education programs. With 
a growing emphasis on learner-centered pedagogy and assessment, teaching and assessing critical thinking have become important 
challenges facing educators. Because much of traditional instruction has focused on the learning of facts and information, teachers 

find themselves facing new expectationsthat require them toachieve different kindsof learning outcomes. In orderto assist educators 

in rneeting the challenges associated with this transition, it is important to provide a framework that provides support and 
clarification, while retaining as much professional autonomy as possible. 

Henry Ford Community College (HFCC) has developed an approach to incorporatingcritical thinking into its general education program 

that seeks to achieve these goals. The main objectives have focused on (1) developing an awareness of the importance of critical 
thinking among faculty and students and (2) fostering a willingness to incorporate the teaching and assessment of critical thinking 
into the curriculum on a widespread basis. 



How Did Critical Thinking Become a Learning Outcome at HFCC? 

In 1996, Henry Ford CommunityCollegebegantheprocessofdevelopinggeneraleducation outcomes. An initialsurveywasdistributed 

to faculty, staff, advisory committee members, and potential employers asking what educational outcomes associate degree 

graduates should beable to demonstrate. As a result ofthis survey, ten broad categories were identified: communication skills, critical 

thinking, humanities/arts, technology and information acquisition, quantitative literacy, scientific literacy, American heritage and 
cultural diversity, human relations/work ethic, health awareness, and lifelong learning. 

This list served as an initial guide to narrow and focus the viewpoints of the college community. The list was distributed to four 
academic areas of the college and to a group of employers and program advisory committees to identify which outcomes were deemed 
essential for degree graduates. Consensus emerged around four major outcomes. They included communication skills, critical 
thinking, technology and information literacy, and American heritage and cultural diversity. These outcomes were subsequently 
adopted by the college senate and college organization as a whole. 

Four campus-wide faculty forums were held to solicit further input from the college community and to develop specific language for 
thegeneral education outcomes. These were approved by thecollege organization in January 1997. Accordingto the HFCC assessment 
plan: 

All associate degree recipients from Henry Ford Community College will be able to demonstrate: 

1. Proficiency in reading and writing in standard English. 

2. Computer literacy i n the retrieval, analysis and evaluation, processing, an d deliveryof information in order to participate 
in a technologically oriented society. 

3. Understanding ofthefoundationsandoperationsof American socialand political institutions and cultureinthecontext 

of a diverse global community. 

4. Critical thinking and problem solving skills in addressing a problem or situation described verbally, graphically, 
symbolically, or numerically. 
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Defining and Assessing Critical Thinking 

The assessment of student learning at Henry Ford Community College has been designed to emphasize faculty involvement and 
control. This reflects the belief that assessment will be more effective and provide more useful feedback to instructors if they are able 

to develop the methods and measures to be used. The assessment of learning, as well as other aspects of learner-centered education, 

represents a paradigm that can sometimes be perceived as intruding on faculty autonomy. Therefore, to increase support for the 
assessment process, HFCC has sought to minimize the degree to which assessment procedures come from the outside. Faculty will 
be more likely to participate in authentic assessment if they develop and control the methods used. 

Henry Ford Community College has a long tradition of shared governance. The organizational model for assessment at HFCC reflects 
this tradition. Assessment leadership comes from the committee for the assessment of student learning (CASL), which consists of 
eighteen elected faculty members, representing all areas of the college, and ten appointed administrators. The committee is co- 
chaired by a faculty member, reassigned from 60 percent of his or her teaching load, and one vice president. The faculty co-chair has 
a highly visible role on campus and, together with a full-time coordinator for assessment, guides and supports the day-to-day 
operations of the assessment process. All assessment procedures are developed by the CASLand approved by the college senate. The 
college organization, made up predominantly of full-time faculty, approves the major components of the assessment plan. 

The first stage of critical thinking assessment at HFCC began with the use of the College Assessment of Academic Proficiency (CAAP) 
test ini998in a selection of courses taken by students neargraduation. However, in 1999, thecollegesoughta broader approach and 
adopted a plan to assess critical thinking in randomly selected courses across the campus. In a policy statement approved by the 
college organization, it was agreed that critical thinking was being taught in all college-level courses, except those in which 
performance (such as certain music or physical education courses) was the primary outcome. As part of this agreement, the college 
began a process in which all courses were to have developed a set of measurable learning objectives, common to all sections of the 
course, that were to include at least one critical thinking learning objective. Over a two-year period, groups of faculty members across 
the college have met to achieve consensus on a set of core learning objectives for the courses they teach, which are then submitted 
for approval to an administrative council. 



The HFCCCritical Thinking Assessment Plan 

lnordertoassistintheachievementofcriticalthinkingoutcomes,acriticalthinkingsubcommitteewascreatedinfalli 999 .Aresident 
philosophyinstructorwho had extensive knowledge ofandexperienceinteachingandassessingcriticalthinkingbecamechairofthis 
subcommittee. As a part of its charge, the subcommittee reported to the larger committee forthe assessment of student learning on 
various critical thinking issues. Members of the subcommittee provided a thorough review of the related literature on standardized 
tests designed to measure critical thinking, forwarded numerous definitions of critical thinking, and created suggestions on how 
faculty would most effectively proceed in their classrooms with critical thinking assessment. 

In January 2001 the subcommittee invited faculty to participate in a pilot program designed to measure critical thinking. Six groups 
volunteered to take part in the pilot program, with the intent that others might learn from their documented processes and 
experiences. Again, in early 2002, another set offaculty members was asked to complete pilot projects. By the end of the pilot program, 
four courses actually conducted studies and submitted assessment reports. They were: Biology 131, Introductory Biology; Philosophy 
131, Introduction to Logic; Math 180, Calculus i; and NSG 255, Nursing and Health Care Systems II. 

Accordingto our general education procedures, which serve as a planning document for the college assessment, critical thinking will 
be assessed in ten courses every year. This past January, the names of courses offering at least five sections were literally thrown into 
a fish bowl for random selection. The committee for assessment of student learning decided to choose one course from each of the 
ten divisions on campus. This way, the assessments would be evenly distributed throughout campus and would facilitate the 
involvement of faculty throughout various disciplines in assessment activities. 

The general education procedures also outline the two-year assessment cycle for critical thinking and the activities that are to occur 
each year. These activities include the following. 



Year One; Preparation and Initial Assessment 

The participating faculty members should complete Steps 1 to 8 during the winter term so that the assessment measure could 
be administered during the subsequent fall semester. Assistance from CASL members will be available for all faculty members 
participating in the assessment process. 
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0 Year One— Winter Term 

1. If necessary, obtain council approval for the course master, thus ensuring that the division has already identified 
common critical thinking measurable objectives. 

2. Create a working operational definition of critical thinking as It relates to the objective to be measured. 

3* Collaborate with all Instructors who will be teaching the course In order to develop a common assignment or test to 

measure critical thinking. 

4. Consult with providers of Instructional support services, if appropriate. In the development of the project. 

5. Ifawrittenassignmentlsused.createan agreed-upon prlmarytraltanalysls(PTA)orrubrlcforevaluatlonofcrltlcalthlnklng. 

6. Each participating faculty will Identify the methods by which critical thinking skills are learned by the students In her or 
his classes. 

7. Develop a preparation guide or supplementary materials, if appropriate, to facilitate successful student completion of 
the assessment. 

8. Prior to Incorporating It Into classes, faculty members developing the assessment procedure will obtain division 
approval. If appropriate. 

0 Year One— Fall Term 

9. Incorporate the assignment or test In all sections of the designated course. 

Year Two: Closing the Loop, Second Assessment and Rnal Reporting 

0 Year Two— Winter Term 

10. The critical thinking review teams will evaluate student work and will report assessment results and recommendations 
to the division/department and CASL. Based on assessment results, faculty members teaching the course will develop 
changes to Improve the teaching or measurement of critical thinking, if necessary. 

0 Year Two— Fall Term 

11. The assessment assignment or test will be administered again In all sections of the designated course. Again, the critical 
thinking review teams will evaluate the results and provide a final report to their division, the committee for the 
assessment of student learning, and the senate. 

As part of their Interim report, due in winterofthe second year, participating faculty members will be asked to record the 
following. 

1. What Is the critical thinking objective you are attempting to measure? This should come directly from the 
course master, on file with the appropriate dean. 

2. Indicate your operational definition of critical thinking as It applies to the objective stated above. 

3. Describe the assignment or test you will administer to students. 

4. Identify how providers of support services were consulted or used in the development of this project. 

5. If a written assignment was used, provide a copy of and rationale for the primary traits analysis or rubric that 
was used. 

6. Identify the teaching methods used by each Instructor to teach critical thinking. 

7. Identify any supplementary materials provided for students to guide them through the assignment. 

8. Describe the results of the evaluation of artifacts. 

9. Describe all Improvements that will be Implemented prior to the next assessment. 

A final report will be due at the end of the two-year cycle that will record any Improvements in overall achievement on 
the critical thinking assessment. 
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Activities to Support Critical Thinking Assessment 

Critical thinking is a phrase that has been used and defined in various ways, in some cases, it is understood to involve the evaluation 
of the thinking of others, as in the analysis of an argument (Cromwell, 1992). In other situations, the term connotes a metacognitive 
process, in which a person consciously evaluates information in order to reach a decision or solve a problem. For example, Norris and 
Ennis define critical thinking as “reasonable and reflective thinking that is focused on deciding what to believe or do” (1989, p. 1). In 
still other cases, critical thinking may simply be linked to higher order thinking skills, as might be found at the upper reaches of 
Bloom's taxonomy-analysis, synthesis, and evaluation. 

Because of the emphasis on a faculty-driven model, HFCC has chosen not to adopt an official definition of critical thinking. Instead, 

a varietyofdefinitionsis made available to faculty members to consider as they develop theirmethodsofteachingand measuring how 

students demonstrate critical thinking. However, to help instructors focus on similar approaches, the committee has recommended 
orienting assessment around four categories of critical thinking: 



Analysis 


To break up a whole into its parts, to identify the relationships among its parts 


Evaluation 


To distinguish between arguments, or solutions to a problem, that are strong and relevant and those 
that are weak or irrelevant to a particular question at issue 


Inference 


To drawa general conclusion by examiningindividualcases and discrimlnatingamongvarlous sources 

of evidence 


Problem solving 


Toexamineaspectsofa problem, identify its causesand consequences, and evaluatevarioussolutions 



These representsuggestionsabout howcriticalthinkingmaybedefined, taught, andmeasured. While latitude is provided to faculty, there 

is also encouragement to pursue critical thinkingin ways that are similar to the broadly recognized views about what it actually involves. 
Additional information and support about defining, teaching, and measuring critical thinking is provided for faculty in other ways. 

0 Meetings and discussion groups. After the random selection of courses for critical thinking assessment, the faculty co-chair of 
the CASL and the coordinator of assessment meet with each team of instructors to provide guidance and assistance. The teams 
are encouraged to meet regularly to reach consensus on how critical thinking would be defined and assessed in these courses. 

Discussion groups are set up for the teams of faculty members whose courses are selected. The purpose of these groups is to 
share across disciplines the ideas, challenges, and approaches the teams have encountered in developing their own approach 
to defining and measuring critical thinking and problem solving. 

0 Workshops. In collaboration with HFCC’s Center forTeaching Excellence and Innovation (CTEI), workshops are offered on various 
aspects of critical thinking. One workshop focused on how to write measurable learning objectives for critical thinking and was 
aimed at faculty members who were developing their course learning objectives. Other workshops address different ways to 
define and measure critical thinking. These include discussions of multiple-choice tests, written assignments, scoring rubrics, 
and related topics. 

0 CASL Website. Ensuringthatinformation is easily accessible is importanttoassistingfaculty.Thecommitteefortheassessment 

of student learning established a Web site that is used to distribute information about HFCC’s assessment policies and 
procedures, how-to guidelines and assistance, forms, bibliographies, and links to many other Internet sites. One section is 

devoted specifically to critical thinking. As teams offaculty complete their reports about their own criticalthinkingand problem- 
solving projects, the reports are also posted on the Web site. Faculty members are directed to the site when questions arise about 
developing their assessment methods. 



What We’ve Learned from the Process 




The process to evaluate critical thinking at Henry Ford is still in the infancy stage, and we hope to identify and reward best practices 
throughout this evaluation process. Already, during the pilot programs, faculty worked closely with the CASL to determine and record 
these best practices. As a result of the first initial group of pilot assessments, the CASL decided to include a second year evaluation 

ofartifactsinordertomeasureanyrealgainsinlearning.Weanticipatethat,astheprojectisfullyimplementedoverthe next two years, 

we will make further modifications to our assessment practices. If necessary, we will modify teaching strategies, testing instruments, 
and assessment methodology in order to ensure that our graduates are meeting critical thinking educational outcomes. 
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Assessing Critical Thinking 
Across the Curriculum 

Nanette Bagstad, Paul Batesel, Ronald Semmens, and MarkSkean 



Since 1992 Mayville State University has been the recipient ofthree faculty development grants funded by the Bush Foundation of St. 
Paul, Minnesota. The grants were designed to improve student learning through faculty development and improvement of teaching. 
Throughout 1992-1995 faculty participated in campus-wide training in cooperative learning; many faculty members continue to use 
cooperative learning in their classrooms a decade later. 



The Need for the Program 

The members of the original cooperative learning governance committee spent approximately two and one-halfyears investigating the 
need to submit a planning grant to the Bush Foundation to study how students could develop skills that would ensure their success 
in the college classroom. The governance committee members also wanted to assure future employers that our graduates possess the 
requisite skills necessary to complete work-related tasks. Discussion among faculty in all academic divisions mirrored the same 
findings: many students who were being accepted to Mayville State University exhibited a diminished level of academic preparation. 
The Liberal Arts Division administered the Nelson-Denny Reading Test to all incoming freshmen and transferees. The results revealed 
that significant numbers of students scored at or below the fortieth percentile in reading. 

Faculty concluded that the reading deficit could be attributed to a number of causes. Most of our students were educated in small, 
rural high schools with very limited course offerings. There was an influx of students who were first generation college students; many 
were from low-income families, and others had been certified as having disabilities. 

The faculty concluded that an emphasis on developing and assessing critical thinking skills would benefit our student body. The 
governance committee members wrote and submitted a grant proposal to the Bush Foundation to create critical thinking activities to 
enhance basic skills in the areas of reading, writing, speaking, mathematical calculation, and computing. The three-year grant, 
entitled Critical Thinking and Assessment Across the Curriculum (CTAAC), was awarded for 1999-2002. 



Critical Thinking Outcomes 

What did we want students to know or be able to do in terms of critical thinking? Since our institutional assessment plan had a critical 
thinking component based on Bloom’s taxonomy, we initially determined that we would use Bloom. However, our association with 
Richard Paul and the Center for Critical Thinking in Sonoma, California, led us to adopt the Elements of Thought as our outcomes. Our 
faculty continued to find Bloom’s categories valuable for creating learningtasks;butthe committeefeltthatthe Elements— question, 
purpose, concepts, information, interpretation, assumption, point of view, and implications— would better prepare our students for 
both college work and the world of work. 



Development of the Test 

Since the primaryconcern bfthis group wasfaculty development, we sawassessment of students as a meansofprovidinginformation 
about our success as faculty developers. We, therefore, wanted a test that would provide accurate information about student 
performance in meeting specific outcomes. We felt that the only way to obtain this information was to make our own test. Based on 
a modelfromtheCenterforCriticalThinking,ourtest consisted ofa reading passage with directed prompts. We selected a short article 
on required summer school from the opinion page of USA Today as a topic that would be familiar to most of our students. 



Development of a Scoring Rubric 

Fifteen faculty members-roughly a third of our faculty-were involved in the test. A rubrics team of eight faculty members from five 
disciplines constructed the test and scoring rubric. Working with a draft of the test, this team went on a retreat in September. They 
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defined the task, took the test themselves, and began to try to determine baseline answers. Discussions were at times heated, but 
by the end ofthe first day, the team had arrived at a consensus about a reasonablestudentresponse. On the second day, the team 
read through a hypothetical set of answers, attempting to decide whether the answers met the baseline, exceeded it, or fell below it. 
Again, after heated discussion, the team decided that the baseline would be workable. 

Duringthefollowing May, the rubrics team met again, this time to extend the baseline to four levels. Again, the team applied the rubric 
to four actual student papers, making revisions they deemed necessary. The grading team arrived, saw the rubric for the first time, 
and tried to apply it to the same set of papers. Their results were in line with those of the rubrics team. The two teams met together 
to discuss problems. After more heated discussion, they revised the rubric again and applied it together to a final student paper. 



Testing and Scoring Methodology 

Freshman and sophomore students were tested during the fall 2000 semester, and those who remained were retested with the same 
instrument during the spring 2002 term. All papers were then retyped and scored by the grading team. This team had been trained 
to perform consistent scoring, and the method of assigning the papers to scorers also contributed to reliable results. Appropriate 
statistical analyses were performed to measure the improvement in each ofthe elements ofcritical thought across the group as a whole 
and for each individual student. 



Results 



A Table of Means, Standard Deviations (SD), t-Test Values, and Significance Levels 
for the Pre-test/Post-test Comparisons of the Eight Elements of Critical Thinking (n = 27) 



Elements 


Pre-test 


Post-test 


t Value 


SIg. 


Mean 


SD 


Mean 


SD 


1 


2.39 


•97 


2.26 


•75 


•574 


ns 


2 


2.02 


.58 


2.00 


.84 


.103 


ns 


3 


2.80 


•59 


2.74 


.70 


•473 


ns 


4 


2.41 


•93 


2.30 


.74 


.465 


ns 


5 


1.98 


•55 


1.85 


•55 


1.000 


ns 


6 


1.64 


.66 


1.54 


.48 


.796 


ns 


7 


2.37 


.64 


2.13 


.64 


1.837 


ns 


8 


1.41 


.77 


1-57 


.81 


.975 


ns 


Total 


17.02 


2.62 


16.39 


2.34 


1.509 


ns 



The exam results provided no early indication of changes in students’ critical thinking across any ofthe eight elements of thought. 

External Evaluator Findings 



The evaluator offered several possible explanations for the flat exam scores. First, implementation activities have yet to be 
institutionalized. Second, the implementations have, in most cases, consisted ofonly one to three lessons covering one to three days 
of individual courses. Third, the students who wrote the posttest constituted a minority ofthe original sample. Fourth, the level of 
exposure of the post-test students to the eight elements was not known. 

The evaluator also offered the following recommendations. 

o Continue the faculty development, faculty instruction, and student learning assessment practices that have been designed 
and implemented during these first three years. 

o Expand the number of faculty members participating in CTAAC activities. 
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o Increase the number of lessons during which an individual faculty member incorporates direct reference to and application 
of Socratic questioning and the elements of critical thought. 

o Intensify student academic experiences with the elements of critical thinking at various meta-cognitive levels, with special 
emphasis initially on identification and recall, so that students can spontaneously name and think about the elements. 

o Apply more control on the student participation variables of the pre-test/post-test assessments. 
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Integrating Faculty Development 
and Outcomes Assessment 

John Neibling, Alan Jacobs, and John Nagy 



Over the past decade, significant progress has been made at hundreds of American community colleges in the area of identifying and 
assessing student learning outcomes. Untold hours of faculty and administrative effort have been applied to the effort to transform 
institutions from the teaching paradigm to the learning paradigm. There is almost certainly an outcomes committee on every campus, 
and there have even been positions created in order to meet the challenge. Accrediting agencies, such as the Higher Learning 
Commission, have provided high levels of support of and guidance to institutions as they have attempted to transform themselves. 
A classic example of such support and guidance is the “Levels of Implementation” document that the Higher Learning Commission 
developed for use by its client colleges, as well as the regional conferences it has sponsored. It would be difficult to overestimate the 
amount of time, energy, and money that has been spent on student learning outcomes assessment. 

In spiteofthe monumental effortthat has been made togetthe paradigm toshift, most institutions still find themselvesstrivingtoward 
achieving the goal of becoming “learning colleges.” One of the reasons that it is so difficult to effect change is that decades, if not 
centuries, of academic tradition, funding structures, governance systems, and even legislative mandates militate against it. 
Therefore, at many colleges, outcomes assessment is viewed as an add-on, rather than a changeover. Processes such as curriculum 
approval, faculty evaluation, and budgeting continue on as they have foryears, even though the institution purports to be committed 
to outcomes assessment. The very existence of an outcomes committee could even be seen as evidence that outcomes assessment 
is not truly integrated into the college's processes. 

While change is often a slow process in higher education, it can happen, especially when faculty members believe that a new idea can 
actually help them in their teaching, rather than beingthe **ideedejour, "as has so often been the case of late. Scottsdale Community 
College has had a professional development system in place for many years, but it was based on the enhancement of teaching, rather 
than the assessment of student learning outcomes. Through the leadership of the outcomes committee, the college is transforming 
its faculty enhancement plan (FEP) from an old-paradigm process into a dynamic assessment vehicle. 

The FEP is the primary vehicle for instructional improvement at Scottsdale Community College. Every full-time faculty member is 
responsible for completing one on a regular basis. For probationary faculty (those in their first five years of teaching), it is an annual 
responsibility. For appointive faculty (those who have taught for more than five years), it is required every third year. At a college such 
as Scottsdale Community College, sixty to eight faculty members might complete FEPs annually, nearly half of the full-time faculty, 
depending on the number of recent hires. While the FEP is primarily a self-improvement tool for faculty, it lends itself very well to use 
as an outcomes assessment tool as well. In fact, one of the questions on the FEP specifically asks how the faculty member will use the 
information she or he obtains from the experience to improve her or his teaching— a goal of any meaningful assessment activity. 

The FEP is particularly adaptable as a tool to conduct outcomes assessment at the course or program level. In order to encourage the 
use of the FEP for this purpose, the college has provided each full-time faculty member with an outcomes assessment guide, which 
includes four models for outcomes assessment that an individual teacher could conduct. The guide provides an explanation and an 
example of each of the four models: (1) the experimental and control group study, (2) the correlational study, (3) the plan-do-check- 
act approach, and (4) the benchmarking study. Having secured the assessment committee's support, the dean presented the idea at 
the next all-faculty meeting, held in January 2002. Although this information was disseminated in the middle of the academic year, 
and therefore the middle of the FEP cycle, thirty-five of the sixty-six faculty members who completed FEPs chose to use the process to 
conduct some sort of outcomes assessment. Through the FEP process, outcomes assessment was conducted at the course level in 
twenty disciplines taught at the college. 

Two FEPs that blended the two purposes of instructional improvement and student learning outcome assessment were done in the 
subject areas of math and biology. In the case of math the faculty member used the experimental and control group model, and in 
biologythefacultymemberusedthecorrelationalmodel.lnbothcases,thefaculty members made extensive use of statistical analysis 
in order to draw their conclusions and make their recommendations for instructional improvement. Also, in both cases, the outcomes 
assessment activities informed their understanding of the needs of their students and the adjustments that they and their teaching 
colleagues needed to make in order to improve the learning of students. 
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In the math example, students were divided into two groups. The first group, the control group, was made up of students who took 
conventional math courses in arithmetic review, elementary algebra, and intermediate algebra. The second group, the experimental 
group, was made up of students who took special sections of the same courses, but whose sections employed an instructional 
innovation called the Maricopa Mathematics Modules. The studycompared the success ofthe two groups in terms oftheir final grades 
in the courses. In the study, throughput success was indicated by successful completion (a grade of A, B, or C) of a math course and 
its successor in two semesters. Throughput rate was calculated as the number of students taking the first course who successfully 
completed thecourse and its successorin twosemesters, divided bytheoriginalnumberofstudentstakingthefirstcourse.The finding 
ofthestudywasthattherewasa42 percent greaterthroughput rate for students in courses usingthe Maricopa Mathematics Modules 
than forthose in the conventional courses. Even more impressively, module students in intermediate algebra completed intermediate 
algebra and college algebra at a rate 59 percent higher than conventional students. 

In the biology example, the instructor studied predictors of success in two biology courses, general biology and human biology for 
allied health majors. The reason these courses were chosen for study was their pivotal place in the curriculum of students aspiring 
to careers in health services. General biology is foundational for students who are required to take courses such as microbiology and 
human anatomy and physiology, as well as many courses taught at the junior level at universities. Failure to learn and retain the 
information in general biology is a virtual guarantee of failure in subsequent courses. Mastery of information in human biology for 
allied health majors is an even more crucial issue in that it is the only general biology course required in programs such as nursing. 
Therefore, it is the only instruction such students will receive in areas such as cellular metabolism, patterns of inheritance, basic 
molecular genetics, ecology, and evolution. Whereas biology majors will have these topics reiterated in higher-level courses, allied 
health majors will not. Therefore, authentic learning in human biology for allied health students is even more essential than for actual 
biology majors. 

In order to understand the learning environment in these courses, the instructor studied the correlations between various factors and 
student success. Such factors included the student's previous college-level science and mathematics training, reading ability, lab 
attendance, and high school attended, as well as the number of hours the student spent working at a job outside ofthe college. The 
instructor then performed a regression analysis on each of these factors and student success. Among the notable findings were that 
students with no previous college math or chemistry training have a significantly higher dropout rate in general biology. Somewhat 
surprisingly, the variables of “high school attended” and “working outside of class” did not yield significant results. There was a 
correlation between being able to understand the textbook and success, and there was a very significant correlation between 
attendance at labs and success in the class. 

While such classroom research is commendable and impressive, it is still subject to the question “So what?” if its results are not 
applied to making improvements in instruction. Accordingly, a question on the FEP report asks instructors how they will integrate or 
apply what they have learned from the process into their classroom teaching. This question essentially closes the loop for the 
instructor and provides the college with specific suggestions about improving Instruction. As faculty members work with department 
chairs in anticipatingtheir participation in the FEP process, opportunities for instructional improvements at the course and program 
levels emerge. In both cases discussed herein, the Information gathered and reported by each faculty member had the effect of 
improving instruction on the individual and department levels. 

In the case of math, it became apparent that the use of the Maricopa Mathematics Modules represented a significant improvement 
in student learning. As a result, those faculty members who teach from the modules believe theirtheoretical position (as implemented 
in the modules) has been validated. They have also felt more confident in encouraging their colleagues to teach from the modules. 
While others in the department have been Interested in the results ofthe assessment, they have remained cautious. Consistent with 
the belief that student learning outcomes assessment is an ongoing, ever-changing process, the college's math faculty members do 
not make the assumption that the Maricopa Mathematics Modules will always result in increased student success northatthey cannot 
be improved. As other members of the department enter into the FEP process in subsequent years, more studies can be conducted 
to determine if the improved learning persists. 

In the case of biology, the instructor was able to make at least two suggestions to the department based upon the FEP. First, it was 
clear that students should be encouraged to take classes in college-level math and chemistry before attempting general biology. 
Second, once in the course, students need to be reminded ofthe essential nature ofthe laboratory experience. In addition, a 
collaboration has arisen within the biology department in which agroup of faculty members teachingcoursesthat require introductory 
biology as a prerequisite are employing a similar analysis oftheir courses to assess how well students perform based on their 
experience in the prerequisite course. 

As colleges move toward the learning paradigm, it behooves them to reinvent existing processes rather than create new ones that 
compete for the time and attention of department chairs and faculty. The faculty enhancement plan at Scottsdale Community College 
is a process that preceded the emphasis on outcomes assessment, and in its earlier form tended to focus primary on teaching inputs 
such as lecture skills and other techniques appropriate to the teaching paradigm. The FEP process began at a time when it was 
assumed that the focus of instructional improvement should be on the teaching inputs, but it has been adapted to meet the challenge 
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of creating a college where the emphasis is moving toward assessing student learning. Although it is only one of many processes at 
the college, its transformation represents a significant step in the process of change. 
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Effective Performance-Based 
Evaluation of College Faculty 



Earl Nicodemus 



Background 

The quality of any college program depends on the faculty members who deliver that program. Yet, institutions of higher education 
often do a poor job of evaluating the members ofthe faculty. Faculty evaluation is threatening to the faculty members, and the system 
used is often the result ofthe campus political process and is soft and cuddly rather than objective and thorough. 

A good faculty evaluation system needs to be influenced by the purpose ofthe institution of higher education involved. For example, 
a technical school's evaluation system would give considerable weight to teaching, since the primary purpose of that institution is 
teaching. A research university would adopt a system that placed much ofthe emphasis on professional activity such as research, 
publication, and participation in the professional dialogue in the discipline. If the role ofthe undergraduate portion ofthat institution 
were more directed toward teaching, then the instrument used to evaluate the undergraduate faculty would be weighted in favor of 
the teaching area. A religious institution might have service as a large part of its mission. If so, then the area of service would be 
weighted accordingly. 

In the mid-1990s, Ronald Zaccari came to West Liberty State College as its president. The institution had a long history of dysfunctional 
faculty evaluation systems. Dr. Zaccari recognized that the faculty evaluation system needed to be based on the teaching, professional 
activities, and service activities of faculty members and challenged the college to develop a system based on those three elements. 
Since the primary purpose ofthe college is undergraduate teaching, the faculty committee initially wanted to put almost all ofthe 100 
points into the category of teaching. After much discussion and work, a system was developed that had 50 points allocated to the 
teaching category and 20 points allocated to each ofthe other two categories. The 10 remaining flex points could be allocated in any 
combination to either or both ofthe categories of professional activities and service. Because ofthe diversity of activities under each 
of these three areas among the departments on campus, each department was allowed to determine its performance factors for each 
ofthe three areas. Another feature ofthe process was a Faculty Performance Planning Document in which the faculty member 
attempted to predict his or her activities in the three areas and defined the allocation ofthe 10 flex points for the coming academic year. 
Although the superstructure ofthe faculty evaluation system provided by the college’s policy was sound, it did not provide for objective 
research-based assessment in the three areas. To accomplish this, the chair of the department of professional education began a study 
ofthe research relating to evaluating college faculty. Research relating to evaluating college teaching was of special interest. 



Evaluating Professional Activities and Service Activities of College Faculty 

Evaluatingteachingwas SO important because ofthe mannerin which the othertwo categories were addressed. In discussingtheareas 
of professional activities and service, the professional education faculty quickly began to list activities and to discuss the relative 
values of those activities. A number of questions were discussed: How much credit does a faculty member receive for a publication? 
Does an on-campus publication get any credit? Does publishing in a non-juried publication earn as many points as publishing in a 
juried publication? What about a chapter in a book? What about a whole book? How about a professional journal compared to the 
newsletterforthe state professional organization orthe regional professional organization? What about Web publishing? Is any credit 
given for maintaining the Web site? Is maintaining a Web site publication or service? How does one determine what constitutes 
professional activity and what constitutes service? For example, where does a faculty member in professional education get the credit 
for conducting a parenting skills workshop for the PTA? Another question that generated discussion was. Can one repeat activities and 
get additional points? How many times? Do I get more points if I go to another state or national conference? What if it is for a different 
organization? When does absence from class to attend conferences become an issue relating to teaching? If I am a state officer, do 
I also get points for attending the state meeting? 

The end result ofallofthis discussion wastheadoption ofa list of prospective activitiesin the areas of professional activity and service, 
with a point valueor point value range assignedto each item. At first glance, thischecklist approach seemswrong. However, it required 
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the department to define the types of professional activities that it expected of the faculty and to determine the weight of the reward 
for participatinginthoseactivities.Untilthelistwas adopted, the evaluators were required to relyontheir professional opinions rather 
than the collective professional opinions of the faculty. In a research-oriented university, the faculty would want to develop very well- 
defined criteria for awarding merit points to each item in the category of professional activities. 



Characteristics of Good College Teachers 

Before college teaching can be evaluated, the characteristics that d efine excellence in college teaching must be identified. Ask anyone 
on our campus who is a good teacher and they will tell you. The same names will be mentioned regardless of who is asked. Ask who 
is a poor teacher, and you will find that students, other faculty members, custodians, and everyone else will identify the same people. 
Ask the students what makes a good college teacher, and they will tell you. Listen to them because they usually get it right. They will 
tellyou that a good teacheris an expertinthe subjectthat he or she isteachingand knows howto explain it so thatthey can understand 
it. They will tell you that a good college teacher treats them with respect. They will tell you that a good college teacher grades fairly. 
A good college teacher does not waste their time. He or she makes sure that the course content is what is expected for the course. For 
example, the content in an American history course should be American history. A good college teacher starts and stops class on time 
and uses time productively. A good college teacher is a good discussion leader and a good questioner. A good college teacher makes 
his or her expectations clear. A good college teacher will go the extra mile to help a student who needs it. Many students will tell you 
thattheyareoffendedwhenteachersuseobscenelanguageandtelloff-colorjokes.Agood college teacheris available to the students. 
Students are quite upset when they visit a faculty member’s office during office hours and find that the faculty member is never in the 
office. They also do not like to be stood-up by a faculty member with whom they have an appointment. Students will tell you that a good 
college teacher is well organized and returns assignments and exa ms to them promptly. A good college teacher uses visuals and other 
materials that are readable and make sense. A good college teacher knows how to make class interesting. A good college teacher 
follows the policies and procedures of the college or university and carries out his or her non-teaching responsibilities. A good college 
teacher works collaboratively with colleagues and others. 

In addition to talkingto the students, this researcher invested many hours in studying research and writings relatingto the evaluating 
of college teaching and the definition of college teaching. No matter how well designed and elaborate the research study or how 
learned the author, the research always identified the same characteristics that the students at West Liberty identified. 



Academic Freedom 

Several of th e items on the aforem entioned list raise questions about academic freedom. Faculty members who teach in our accredited 
teacher education program do not have much real academic freedom. A better term might be academic latitude. The professional 
education program at West Liberty is a carefully planned sequence of courses and experiences designed around an array of local, state, 
and national professional standards. Each course in the sequence has a specific purpose and includes a well-defined portion of the 
knowledge base for a beginning teacher. The faculty member teaching the course is responsible for ensuring that his or her section 
of the course includes those items. The faculty members plan collaboratively to ensure that all sections of a given course include the 
same essential content. Field experiences and similar activities associated with a course are designed collaboratively so that the 
expectations are the same for all sections. Each faculty member has some professional latitude in regard to his or her teaching style, 
but must includethe agreed-upon content and experiences. Thus, workingcollaboratively with colleagues is an important expectation 
of our faculty members. 



Sources of Data 

□ Student Evaluations 

A lot of controversy still surrounds the use of student evaluations of faculty. However, a large body of research supports the 
validity of student evaluations provided that the right questions are asked on the evaluation form. Rather than repeat a well- 
researched argument here, I referthe readerto a couple of good sources of information on the subject: University of Massachusetts 
Ikmheist Assessment Bulletin, Octoberi995<http://www.umass.edu/oapa/assessment/ANewDesignforUMass.pdf>and“WhatDo 
They Know Anyway?” Richard M. Felder, Professor of Chemical Engineering, North Carolina State University <http://www.iub.edu/ 
~teaching/felder.html>. Professor Felder’s paper includes a large number of references to documents pertaining to student 
evaluations. 

The questions presented on student evaluations need to be about things that a student should be able to rate. Examples include 
“The instructor appeared to know his or her subject”; “The instructor made good use of class time”; and “The professor treated 
the students with respect.” 
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If the student evaluation form includes questions relating to the course rather than to the instructor, those questions should not 
be used in evaluating the instructor. An example of such a question is “Did you think this was a worthwhile course?” If the student 
evaluation form is properly constructed, the faculty member can be provided with an average rating andean be informed of how his 
or her average rating compares to the average ratings ofother instructors forthesame course or of all ofthe instructors ofthecourse. 

In addition to the rating list, most evaluation forms include a space for students to write comments. The student comments are 
a useful source of data relating to the quality ofthe teaching. Any comments relating to the course rather than to the instructor 
cannot be used to evaluate the instructor. An example of such a comment is “SPED 440 should not be required for special 
education minors because we have already had all of that stuff” (an actual student comment!). After removing all comments that 
do not relate to the instructor's teaching, the remaining comments can be rated as negative, neutral, or positive. Then, one can 
determine the percentage of negative, neutral, and positive comments by simply counting them. One must emphasize that 
student evaluations are only one source of data. Although the research supports their use, an evaluation system that relies only 
on student evaluation data is flawed. 

□ Peer Reviews 

Peer reviews provide another useful source of data. However, peer reviews are difficult to carry out in an objective manner in small 
institutions. The results tend to be inflated most of the time, with notable exceptions when the peer reviewer has a personal 
dislike for the faculty member. Our institution has used peer reviews in the past, but does not do so today for these reasons. 

□ Class Visits 

Class visits by administrators or peers are useful if they are well structured. We use a form that allows the visitor to record what 
is taking place in the class every five minutes. It is in the form of a matrix, with the time across the top and a list of possible 
activities down the left side. It also includes a section that allows the observer to record the number and types of questions asked. 
Another line asks the observer to rate the activity in regard to teaching critical thinking. The observer also rates the appropriate- 
ness ofthe content, the level of student involvement, and the interest level during every five-minute interval. The bottom ofthe 
form has space for comments. At the conclusion ofthe observation, the observer prepares a report to the instructor. Class visits 
tend to be quite positive and usually provide data that is complimentary to the faculty member. 

□ Peer Comments and Complaints 

As a department chair, 1 receive both complaints and positive comments from faculty members about each other. These must be 
considered holistically. In a small college, petty feuds and quarrels sometimes create negative relationships. As the combatants 
carry out their little wars against each other, they like to file complaints or to register criticisms of each other. Sometimes, they 
will try to enlist recruits from among the faculty to support their side ofthe quarrel. I think every campus has these little wars. 
One must examine every cannonball that is fired. Sometimes they are on target, and sometimes they are just so much noise. 
However, a pattern of complaints by a cross-section of faculty members pointing to a specific problem in the performance of a 
faculty member is another matter. To be used in the evaluation process, specific evidence must be provided. A faculty member 
cannot be negatively rated on an evaluation just because another faculty member doesn't like him or her. 

□ Comments and Complaints By Students, Parents, and Others 

In our department, complaints from students, parents, and others are evaluated, and those that may have merit or that may need 
further action are recorded, placed into the file, and reported to the faculty member. As with comments and complaints from 
peers, most individual complaints are not used as evaluation data. However, a pattern of complaints about the same problem 
is used. Any individual complaint involving serious misconduct by a faculty member would be reported to the appropriate 
institutional authorities for investigation. Most comments that are made by students, parents, community members, and others 
are positive. Negative comments usually relate to grading or to respect for the students. 

□ Review of Course Materials 

A review of the materials used in the course provides data relating to the appropriateness of the content, the fairness of the 
grading system, critical thinking, and other matters. 

□ Records of Absences and Tardiness 

Faculty members miss classes for a variety of good reasons. However, a pattern of poor class attendance or of tardiness is 
unprofessional and should be reflected in the evaluation ofthe teaching. 

□ Course Enrollment Data 

This is controversial. Should low enrollment reflect negatively on a faculty member’s evaluation if other sections ofthe same 
course have much higher enrollments? A few years ago, in one ofthe departments on our campus, students nearly all dropped 
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a course by a faculty member before the end of the term because they felt his grading was unreasonable. Other sections of the 
same course were full to the limit. There was considerable resentment among his peers because they had full classes while he 
was teaching to two or three students. 

□ Grading Patterns 

Should the grading pattern of a faculty member be a factor in evaluating his or her teaching? One former faculty member gave 
every student in every class an A. Another almost never gives an A. Is a grading system fair if the students who do not do the work 
get the same grade as those who do? An abnormal grading pattern may indicate a flawed grading system. In reviewing grading 
data, one should consider the nature of the course, because some upper level courses have very high levels of student 
achievement that are reflected in the grades. 



General Comments 

Our department has adopted a rubric that lists the items mentioned in this document and provides descriptors and point values for 
them. By breaking the teaching category down into small elements and then evaluating each element, a more objective evaluation is 
possible. An evaluator must learn to set aside his or her personal feelings and to rate the faculty member fairly on each component. 
An excellent faculty evaluation system is possible if we clearly define what we want the faculty member to do and then design a rubric 
to measure whether it was done. 



Earl Nicodemus is Interim Chair of the Department of Professional Education at West Liberty State College in West Liberty, West Virginia. 
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Nancy Blackford and Kate Wiles 



Using assessment initiatives in education at Walsh University as a model, this study explores the means of balancing data-driven 
program review, new content standards, and state licensure requirements while preparing teaching professionals. It helps readers to 

o Understand the contexts that propel data-driven decision making, particularly in teacher preparation 
o Ask critical questions related to institutional and program assessment 
o Involve an increased number of stakeholders in the assessment process 
o Design a data management system suited to their unique needs 
o Use data to inform instruction and program improvements and to demonstrate quality 



The Story of Education Accreditation at Walsh 

Walsh University is an independent, coeducational. Catholic, liberal arts and sciences institution located in northeastern Ohio. This 
Masters I university enrolls about 1,800 (headcount) and offers forty-seven undergraduate majors and four graduate programs. More 
than 400 undergraduate and graduate students pursue teacher preparation within the Division of Education-about one-third of 
Walsh's full-time population. The university is appraised by associations such as the Higher Learning Commission and the Council for 
Accreditation of Counseling and Related Educational Programs (CACREP). Recently, the Ohio Department of Education revised its 
requirements for all teacher education units in the state. Each professional education program must now meet the 2000 National 
Council for Accreditation of Teacher Education (NCATE)criteria,therebyintensifyingrequirementsthatthe division collect, aggregate, 
analyze, and apply empirical data so as to inform program improvements, provide systematic feedback to students, and monitor 
candidate progress toward initial teaching licensure. Walsh University was the first teacher preparation institution in the state to 
achieve five-year approval according to these criteria. 

The Ohio Department of Education (ODE) granted this approval, in part, because ODE representatives said the assessment plan 
exceeded their expectations and those of NCATE. Two documents from the division, the conceptual framework and the assessment 
plan, are now being widely examined as models around the state and even in neighboring states; and faculty from the Division of 
Education have been invited to help train other institutions about the newly revised standards. 

The conceptual framework developed by the Division of Education articulates three fundamental commitments: to diversity, to judeo- 
Christian traditions, and to service learning. These guiding principles link the work of university faculty, pre-service teachers, and in- 
service educators who collectively form the “Connected Community” of the unit's logo. During the recent accreditation process, this 
aspect of the unit's framework was specifically commended. Additional stakeholders contributing to the successful implementation 
of this data-driven model include administration, faculty in other subject area divisions, the Consultative Council (a group of present 
and past educators from area schools), and informational technology personnel. 

The Master Plan of Assessment of the Division of Education requires both internal and external data collection at five points in a 
candidate's progress toward initial licensure: 

Point A: Admission to the university 

Point B: Admission to the teacher preparation program 

Point C: Completion of the pre-clinical coursework 

Point D: Degree and licensure completion 

Point E: Entry employment year(s) 



According to the state, the initial assessment plan submitted with the self-study was deficient. In the rebuttal that Walsh had 
opportunity to prepare during the thirty days after the team's visit, the division transformed the initial sketch into many flow charts 
on legal-size paper. The board still considered the plan insufficient and planned to award two-year approval-otherwise described as 
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at-risk status. With one-third of the undergraduate enrollment in education at stake, the university’s president instructed his staff to 
request a hearingwith the ODECommission, which resulted in an invitation to the universityto submit a third version of itsassessment 
plan. With able assistance from a technical adviser assigned by the state, the division added narrative, more-complete charts, and 
more-readable tables. This seventy-six-page version brought five-year approval alongwith commendations as an exemplary plan. This 
case study outlines the collaborative, developmental process that produced the data-driven paradigm now guiding assessment and 
decision making during the pilot year of implementation. 

Intellectual and Cultural Contexts 

To understand such intense scrutiny ofdivisionsofeducation as one manifestation ofongoing national quality initiatives, this written 
portion of the study briefly examines cultural contexts fueling demands for outcomes-based measures of teacher preparation 
programs.The “discovery” oftheNewWorld in the early Renaissance and the foundingofthe United States in the Age of Enlightenment 
endowed American culture with enduring faith in reason and mechanical principles. The Industrial Revolution advanced mechanistic 
thinking, even while it was counterbalanced by the Romantic and Darwinian movements in the nineteenth century. The development 
of the American educational system reflects this heritage-the belief in the individual’s ability to reason and in the civic necessity of 
training the masses. Cultural influences of the twentieth century deepened and broadened this heritage. Normal schools were born 
at the time of Dewey’s emphasis on pragmatism, Henry Ford’s division of work into simple and discrete repetitions to be learned by 
the masses, cadres of efficiency engineers who followed Deming, and the deeply rooted tradition of logical positivists, rational 
idealists, and behaviorists. Such forces focused twentieth-century American education on the linear and the hierarchical. 

The culture wars of the later twentieth century saw cumulative challenges to these principles: Einstein’s concept of relativity and the 
demise of Newtonian physics, tides of philosophic and literarytheoryfrom Europe, waves of cognitive research,the sexual revolution, 
the Vietnam War, the shrinkingofthe globe, and the changingcolor of America’s and the world’s body politic. Education changed. The 
Montessori movement, thedrive for relevancein thecurriculum,theopen classroom, altered dress codes, thegrowth of special education, 
multiculturalism in textbooks, bilingual education, and the integration of extracurricular activities were some of the changes. 

The changes provoked resistance. Works by Bloom {Closing of the American Mind), Hirsch {Cultural Literacy), Bennett {To Reclaim a 
Legacy), and S\m{Horace*s Compromise) fanned state and local debate about the state of American schools. In the 1980s and 1990s, 
corporations seemed to be models for correcting the permissive “relevance” of the 1960s and 1970s and for restructuring to achieve 
lean management. Bit by bit, basic, solid knowledge ofthe pastwas made a bulwark against revisionism and theory. Manytookcomfort 
in learning that continued to be “a predictable, mechanical process measurable by observable responses, which led teacher 
education to focus on lessons and lesson planning that required bits and pieces of skills...based on assumptions ofthe linearity of 
learning” (Blackwell, p. 363). The people, the state, and the nation would know whether schools were doing the job. Teachers would 
be tested; students would be tested; programs would be measured. Accordingly, approval and funding would follow, or not. 

In the context ofongoing debate, the demand for accountability in education-actual performance-based results— gained voice and 
credibility. Broad recognition of Alverno College’s competency-based curriculum, the great success of Angelo and Cross’s guide to 
Classroom Assessment Techniques {1993), and changing criteria mandated by accrediting bodies such as the Higher Learning 
Commission showed a national consensus calling for assessment plans and aggregated data as measures of good practice. The 
transformative power of assessment began to emerge. As of 1995 all Higher Learning Commission institutions were required to have 
a comprehensive assessment plan in place. Ongoing reviews now require analysis of patterns of evidence supportingthe program’s 
continuation and growth. 

As data-driven decision makingspread throughout state, regional, and national contexts,Ohiobecame a particularly interestingstudy 
in itself. While weathering a decade of litigation on vouchers and the constitutionality of funding schools via property taxes in the 
1990s, the Ohio Department of Education responded to Title II ofthe Higher Education Act for approving licensure and teacher 
preparation programs by becoming one ofthe first in the country to adopt NCATE standards in 1998. In 2002 Ohio was one of thirty- 
two states with standards in place or in process; the other eighteen states had not yet set standards. Against this broad backdrop, 
this case study demonstrates how one model assessment plan-deemed satisfactory by the Ohio Department of Education and 
embraced by the division-now grounds the unit’s decision making in fundamental commitment to gathering, aggregating, and 
analyzing data in order to discover patterns, decipher strengths and weaknesses, and make appropriate changes based on evidence. 



Critical Questions for All Education Divisions 



Question One: 
Question Two: 

Question Three: 
Question Four: 



What are the standards by which the division will be measured? 

Since the measures must be holistic, field-based, and performance-based, how does a unit authentically 
measure itself against those standards? How does a unit do this with validity? 

How can the unit manage thousands of data bytes and glean meaningful information? 

How can the unit apply the data to inform decision making, to make genuine improvements, and to 
demonstrate quality? 
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These basic questions are not simple. As an Ohio institution Walsh must meet ten cofepor/es of professional performance described 
by the Interstate NewTeacher Assessment and Support Consortium (INTASC) Standards. These aspects of candidate performance are 
classified according to subject matter, student learning, diverse learners, instructional strategies, learning environment, communi- 
cation, planning instruction, assessment, reflection/professional development, and collaborations, ethics, and relationships. 
Additionally, the Walsh assessment plan must confirm that the candidate has mastered essential elements of professional practice 
as defined by the Praxis/Pathwise model. Ihe four domains of this model constitute evaluative criteria that state assessors will use 
during the candidate’s first years of teaching. Thus, the preparatory program must also measure each pre-service teacher’s 
performance against the standards. 

Finally, the assessment plan generates data to attest to the candidate’s successful performance in terms of university and Division 
of Education standards. Academic achievement and comprehensive studies in general education are required of all program 
completers. Further, the division’s conceptual framework requires of its programs, faculty, and students commitments to diversity, 
service learning, and judeo-Christian traditions. Thus, the assessment plan must provide evidence related to these three essential 
attributes of candidate performance as well. 

Utilizing three sets of standards (NCATE, Praxis/Pathwise, and the Walsh conceptual framework), the master plan of assessment 
specifies sources, timelines, instrumentation, and evaluative criteria for collecting data. It also outlines feedback loops for individual 
pre-service teachers, the division, and the university at large. 

While the fundamental process of data gathering and aggregation may suggest a mechanical series of checklists, the complex 
standards for measuring students, teachers, and programs are Informed by current theory. Holm and Horn (2003) have cross- 
compared the standards of NCATE, INTASC, and the National Board for Professional Teaching Standards (NBPTS). They summarize 
these standards as 

o Knowledge and skills that will help [teachers] know and understand who their students are and how they learn 

o Knowledge and understanding of the content of the disciplines and of the Instructional strategies that can be used to create 

powerful learning experiences 

o An understanding of the role of assessment and how to design assessments that will inform practice and guide student 
learning 

o Habits of reflection 

o A collaborative approach to their work with colleagues, families, and communities (Holm & Horn, 2003, p. 377) 

These criteria require relational learning communities in which discovery and experiment can occur. They imply incremental growth 
that Is dynamic and cumulative. Data are gathered to verify growth and to benchmark the achievement of standards, but the larger 
target Is holistic efficacy of the individual teacher and student and of the program, university, or community In which they function. 
ThegoalsoftheDivIslonofEducation’s assessment plan at Walsh University are both to meetthe accrediting bodies’ need forevidence 
and to make evidence-based, data-driven decisions that genuinely inform best practice. 



Stakeholders’ Commitment 

Throughout the accreditation and assessment efforts, the division adopted a team-driven process to create its conceptual framework 
and the assessment plan. No person on staff was specially trained in assessment; the state of Ohio was not able to offer models of 
either required document. While the university as a whole was deemed to be at Level 2 during its 1999-2000 comprehensive review 
bytheHigherLearning Commission— a reviewthatsetthetimingofthenextcomprehensivereviewintenyears— Walsh has never been 
a member of NCATE. Moreover, during the very months the division was drafting its framework and self-study, state department 
personnel were themselves coming to an understanding of the criteria and were defining processes for reviewing schools. The 
conceptual framework has gone through one post-review revision to date; the assessment plan has gone through eight substantive 
revisions. 

Several factors contributed mightily to stimulatingand maintainingthe team effort. The divisional chair played a major role. The chair 
changed in july 2001; Walsh was reviewed in October 2001. But both the outgoing and the incoming chairs modeled constructive 
attitudes, keeping morale and commitment very high. Administrative support was strong-with some new personnel hires, with 
refurbishing ofthefacilities, with editorial assistance, and with stipendsfor writers. Thefacultyshared the load, meetingoncea week 
for one crisp hour and monitoring themselves for productivity-in part, through the unit’s stellar minutes. All individual full-time 
faculty reviewed each draft of the plan and gave feedback. Cooperating teachers were involved through special preparatory meetings 
and an open forum on the assessment plan. The Consultative Council was revitalized with meaningful responsibilities and authority; 
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community partners therefore were willing to give useful feedback on the plan. The ODE technical assistant was regularly available 
by e-mail, by phone, and in person. These collaborators helped guide the division's deliberations. The president's support in the end 
was the absolute bedrock, for he insisted that the mission and welfare of the institution required petitioning the state to reconsider 
the assessment plan. 

Without the active, direct involvement of all these stakeholders throughout the process, the worth of the assessment initiative would 
have been seriously compromised. Like most small institutions, Walsh does not have time, finances, or personnel to engage in 
assessment for its own sake. To justify the sacrifices entailed in the assessment plan, results had to be comprehensible, credible, and 
valid— particularly to those directly affected. Stakeholders had to feel empowered to assist in designing, revising, implementing, and 
utilizingthe plan. This was possible only by including stakeholders at every point in the developmental process. Studentsthemselves 
gave feedback to rubrics during the pilot year's first semester through the Education Club, the education honorary, and individual 
classes. 



Data Management Design 

Using the adopted standards of professional practice (as defined by INTASC, Praxis/Pathwise, and the conceptual framework) to 
identify what\Nas to be measured, the assessment team was now ready to confront the practical issues. How? How often? By whom? 
What data bits were worth using? How might the unit glean, store, aggregate, and interpret these byte sets? Assessment options were 
limited by physical storage space, technological capacity, and the challenges of having part-time team members working on this 
burgeoning project. The limits were fortuitous in prompting use of some data already regularly produced and available. 

For the first assessment “gate,” the admission to the university (Point A), it was readily evident that records kept by the Admissions 
Office and Information Technology department included instructive data elements: high school GPA, SAT/ACT scores, placement tests 
in English and math, and the California Critical Thinking Skills Test. 

Defining Point B (admission to the teacher preparation program) data elements was more challenging. What instruments would 
produce data relevant to each of the ten INTASC standards? The division met to identify assignments or activities in four required 
introductory education courses that specifically addressed these standards. To demonstrate proficiency, each pre-service teacher 
(regardless of licensure track) would satisfactorily complete at least two performance-based activities per standard. While most 
instructors were already using rubrics to scorethese assignments, the next challengewasstandardizingthe assessment instruments 
across multiple sections of the same course, among various instructors, and between courses. This required appreciable staff 
development and collaboration. 

Once the exponential size of available data was fully grasped, the next key decision was to focus on specific data elements in the 
rubrics correlating precisely with an INTASC standard or Praxis/Pathwise domain. The three core components of the conceptual 
framework Oudeo-Christian traditions, diversity, and service learning) also required similar identification of assessment focal points. 
The developmental portfolio was chosen to measure in a formative way the candidate's fulfillment of each standard at intersection 
of Points B and C. 

As with Point B, discrete data focal points were identified for Point C (completion of pre-clinical coursework), this time using upper- 
level education courses to confirm that all defined best practices (according to INTASC, Praxis/Pathwise, and the unit's conceptual 
framework) were systematically assessed during each candidate's program of studies. As a result, through Points B and C, candidate 
performance on each of the ten INTASC standards is assessed at least four times prior to the student teaching semester. Additionally, 
discrete data are now available for all elements of the Praxis/Pathwise model and the core components of the conceptual framework. 
The professional portfolio documents in a summative way the candidate's fulfillment of all criteria, as measured at the intersection 
of Points D and E. Point D (program completion) data points are now beginning to accrue— most significantly. Praxis II scores. Praxis 
II tests are nationally normed tests required by the division and by ODE in licensure areas and on principles of learning and teaching. 
Point E data will begin to accrue in fall 2003 as candidates take their first teaching jobs. 

In summary, major components ofthe assessment plan are admission data; performance in theyears priorto student teachingon core 
components of four courses common to varied tracks, as measured on rubrics adopted throughout the division and assembled in a 
developmental portfolio; performance in the student teaching terms, as measured through division-wide rubrics and assembled in 
a professional portfolio that is evaluated by internal and external assessors; performance on varied national Praxis tests; and 
performance as a practicing professional. The division has identified particular rubrics, tests, and measures as key performance 
indicators (KPI's) that will be aggregated and analyzed at least annually as a means of deciphering trends, gaps, and strengths that 
might then be continuously improved. 

In this pilot year, many aspects ofthe assessment design are still being modified. But already, the students, faculty, and teacher 
preparation program itself are benefiting from both the developmental process and the evidence generated thus far. 
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Evidence and Quality Indicators 

The unit has evidence of maintaining systematic and coherent focus on articulated objectives. The evidence is the standardized 
rubrics, whose development prompted much discussion about course objectives, about new ways for some instructors to deliver their 
courses, and with adjuncts about the division's goals and means of measuring success. On matrices recording data from rubrics, the 
unit is just beginning to aggregate. Despite some anomalies, there is, on the whole, respectable consistency among means and 
medians in the data gathered for Points A, B, and C. Points D and E data are pending, as of this writing. 

One example of the data collection, aggregation, and analysis underway focuses on the INTASC standard of communication. At Point 
A, verbal scores from ACT/SAT tests for declared education majors are compared against overall entering class scores. Additionally, 
the performance-based writing assessment administered bythe English departmenttoenteringfreshmen provides comparative data. 
At Points B and C, education students' developmental portfolios now include elements focused specifically on oral and written 
communication skills. The unit's comprehensive matrix of these data elements allows analysis by student record (throughout the 
candidate's preparation program; by standard (in this case, communication as discretely evaluated in eight separate assessment 
opportunities); by course; by section; and by specific assignment or activity. The essential professional skill area of communication 
is further assessed at program completion (Point D) through the development and submission of a professional portfolio. During the 
successful candidate's early employment years, assessors from the Ohio Department of Education will use the Praxis III instrument 
to evaluate the new teacher's communication skills throughoutthefourdomainsofprofessional practice. These data will beforwarded 
to Walsh University and the Division of Education for further analysis and as evidence of program efficacy in all areas assessed. 

A second example of the unit's approach to data collection and analysis addresses attention to student diversity, which is identified 
by INTASC, Praxis/Pathwise, and the unit's conceptual framework as critical to successful teaching. Beginning with the student's 
admission to the teacher preparation program and through the completion of the developmental portfolio, each candidate must cite 
evidence of varied field experiences in terms of student age; identified special needs; and school type (rural, urban, suburban, 
parochial). Additionally, through required assignments of varying sorts, each candidate must demonstrate sensitivity to diversity 
issues and ability to modify instructional practice for optimal accommodation. Each activity provides a data collection point, 
confirming explicit attention to diversity and measuringthe quality of candidate performance. In addition to informing students about 
the importance of diversity in professional preparation, initial findings have already affected the unit. While initial pilot data were 
readily available concerning students with disabilities, other aspects of diversity (e.g., multiculturalism, socioeconomic status, 
language) rarely surfaced. The division has already modified course requirements accordingly. Thus, attention to diversity is now being 
documented within university course requirements, required field experience activities, and the summative portfolio. At each point, 
the assessment plan defines precise data to be collected, aggregated, and analyzed. 

The unit anticipates continuing to meet regularly and staging workshops for both full-time and part-time faculty to ensure consensus 
in interpreting standards, protocols, and findings. A mentoring group instituted in spring 2003 will specifically address the needs of 
adjunct faculty in this data-driven model. Other activities are planned to sustain the involvement of multiple stakeholders in the 
process. For instance, to produce data attesting to inter-rater reliability, each developmental portfolio submitted by a pre-student 
teaching candidate was assessed both by the unit instructor and by a member of the Consultative Council. Such initiatives 
demonstrate the Division of Education's commitment to using data to inform professional practice. Adjustments will continue as this 
systematic process reaches full implementation, thereby demonstrating that all program completers model best practices. 



Future Implications 

Walsh’s successful example of reaccreditation within shifting standards has relevance for the constituencies of the Higher Learning 
Commission. Colleges continue to struggle with finding alternatives to NCATE (e.g., TEAC, Teacher Education Accreditation Council); 
they wrestle with increasing emphasis on accountability in education (e.g., the national Title II Report Card). State departments of 
education realign their processes and benchmarks, and accrediting bodies under heavy scrutiny periodically reexamine their role and 
expectations— regional accreditation out of Chicago now offers institutions the Academic Quality Improvement Project option. 

National scrutiny of teacher preparation continues to undergo profound change. The 2001 No Child Left Behind Act and, directly on 
its heels in Ohio, Senate Bill 1 both place axiomatic emphasis on testing and on data-driven decision making. Ohio is now one of only 
five states with accountability plans that comply with the federal No Child Left Behind Act. Within such a context, one small private 
university in a rust belt state has so far successfully navigated reaccreditation in this dynamic era. 
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Capitalizing on Internal Talent: 

A Model for Professional Development 

Alex Birkholz and Renelle Gill 



Introduction 

As financial realities hinder the ability to providequality Staff development, the solution may come from within. Wisconsin indianhead 
Technical College (WITC) is a postsecondary institution located in northwest Wisconsin. WITC is a member college of the Wisconsin 
TechnicalCollegeSystem. With a designated geographic service area ofio, 000 square miles, WlTCoperates in an area larger than nine 
of the United States. The eleven-county technical college district is served through four comprehensive campuses in the cities of 
Ashland, New Richmond, Rice Lake, and Superior. Additional staff members work at branches and learning centers, and in the 
administrative offices in Shell Lake. 

WITC had a history of providing extensive staff development through the use of outside experts. Overtime, the direct costs as well as 
the logistics of serving staff at multiple locations caused the college to investigate this practice. The very geography of the district 
presents an unusual challenge in providing staff development. Employees may have to drive more than three hours to participate in 
an activity offered at a college location other than their own. 

WITC offers associate degrees, technical diplomas, and continuing education courses for those seeking new employment or 
broadeningtheirskills in an existing job. Instruction focuseson the areas of agriculture, business and marketing, emergency services 
and health, general education, industrialtechnology, and personal and community service. The faculty and staff supporting this wide 
variety of educational programming possess a tremendous amount of talent. In addition, years of train-the-trainer type seminars from 
outside clinicians had prepared internal staff to deliver staff development in many areas. There was a large pool of resources right 
within our institution. 



The Learning Academy Becomes a Reality 

The Learning Academy was developed asamodeltoprovideprofessionaldevelopmentopportunities to promote, develop, and expand 
the quality education and services of WITC. From humble beginnings in 1999, the Learning Academy has expanded to utilize and 
enhance the talents of staff at all levels of the organization. Success has been measured through participation rates and participant 
assessment of their experiences. A large part of the Impetus for the Learning Academy comes from a movement toward continuous 
skill enhancement for all staff. Everyone is a teacher and a learner. The Learning Academy helps this vision come to fruition. WITC has 
embraced a learning college model, and the Learning Academy is a natural outflow from that model. 



The WITC Learning Academy 

The Learning Academy provides professional development opportunities to promote, develop, and expand quality education and 
servicesofthe Wisconsin Indianhead Technical College. It exists to encourage the success and achievementofWITCstudents, faculty, 
and staff. This specifically includes development of custodial, office and technical support, faculty, and management with specific 
learning outcomes designed for each group. 

The following is an example of the learning outcomes for the faculty group. The Learning Academy strives to provide opportunities for 
faculty to develop into individuals who 

o Are active, lifelong learners who will assume a confident, effective, and responsible role in the world of work 
o Assess their own learning and progress toward established goals and outcomes in personal and professional life 
o Make decisions using critical thinking and problem-solving skills 
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o Apply social interaction skills to develop productive, respectful relationships 
o Effectively communicate information in personal and group settings 
o Demonstrate proficiency in areas of responsibility 

o Demonstrate proficiencyin the planningand strategic implementation ofauthenticlearningexperiences that meetthe needs 

of diverse learners and promote critical thinking and creativity 
o Demonstrate proficiency in documenting and assessing student learning 

The development offerings of the Learning Academy must work toward the desired outcomes of the employee groups. Recent offerings 
include development opportunities in 

o Academic advising 
o Online course server software 

o Licensure and certification 

o Instructional television training 
o Software packages 
o Performance-based instruction 
o Web site design 

o Facilitating the future (instructional design, methodology, assessment, and organizational transformation) 
o Learning pathways (instructional design, delivery, and assessment) 



The Logistics 

All offerings of the Learning Academy are coordinated through a professional development specialist. This position serves to provide 
logistics, strategic planning, needs and outcome assessment, and communication of all Learning Academy activities. The position 
coordinates communication oftheLearningAcademy facilitators. These facilitators guide participants through the staff development 
activities. Facilitator meetings are used to discuss successes, improvement strategies, and the expansion of offerings. 

The Learning Facilitators come from the WITC staff. Members possessing talents in areas supporting staff learning outcomes are 
invited to propose development opportunities. Staff members then facilitate learning for their co-workers and serve as mentors. 
Through this process, existing expertise is honored, and those wishing to expand their abilities are encouraged to take on a facilitator 
role. 

Upon receipt ofthe Learning Academy course catalog, WITC staff members register through the office of the professional development 
specialist. The development opportunities utilize the Internet, face-to-face meetings, instructional television, team teaching, and 
various other methods to deliver learning opportunities. 



Profile: Learning Pathways 

Learning pathways is a major offering of the WITC Learning Academy, and more than 160 faculty members and managers have 
participated in this four-weekend development opportunity. Learning pathways prepares participants to apply the principles of 
accelerative learning to the classroom. The concept of preparing a positive learning environment through the use of music, 
peripherals, manipulatives, and suggestology is explored. The principles of brain-based learning, constructivism, multiple intelli- 
gences, assessment, and learning styles are incorporated and applied in the learning process. 



Profile: Facilitating the Future 

Facilitating the future reflects the need for changes in educational practices that foster administrators*, teachers*, and students* 
abilitiestothinkcritically. More than 900 pre-K-16 faculty members and managers have participated. Daily learningincludes a series 
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of experience-based activities. Experiences allow active inquiry into living and learning to increase learners* resources while 
promotingthegrowth ofthelearningcommunity. This offeringis a partnership between WITC and twoWisconsin cooperative education 
service agencies. 

This project, beginning its fourteenth year, is designed to be a continuous, long-term staff development experience. The program 
includes a two-week summer learning community experience with options to continue that learning throughout the academic year. 
Each community is designed to create synergy between the participants. Typical learning community themes include critical thinking, 
designing learner-centered classrooms, assessing student learning, and cultivating creativity. 



Conclusion 

The WITC Learning Academy has provided a framework through which to offer staff development. In addition, the academy has 
provided opportunities for staff members to share their existing talents and learn new skills. This method of capitalizing on internal 
talent is in keeping with the philosophy that everyone in the organization is both a teacher and a learner. 



Alex Birkholz is Progrom Director for Marketing and Retail Management at Wisconsin Indianhead Technical College in Nevy Richmond. 
Renelle Gill is Progrom Director for Marketing at Wisconsin Indianhead Technical College in Rice Lake. 
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Developing the Capacity for Synthesis: 
Demanding and Assessing for 
High-Level Learning 

Carol Cahavan 



Overview 

As college faculty we have received little in the way of systematic development in the areas of designing courses based on high order 
learning objectives for students. Even less assistance has been available in developing effective assessment strategies. Conse- 
quently, we have been accused of “testing what is easy, rather than testing what is important” (Grant Wiggins, personal communi- 
cation, 1997). College teaching and learning should provide opportunities for students to synthesize components of a course or 
curriculum into new understandings and perspectives. Nevertheless, examples of learning activities that demand little more than 
memorization and analysis are plentiful. Are there ways to ensure that we are designing courses that will demand this high cognitive 
level of achievement, and are there tools to help us in our assessment of synthesis? Previous research (Canavan, 2001) explored these 
topics through the examination of senior-level interdisciplinary courses (referred to as Tier HI) at Ohio University. To assess student 
products I used the work of Biggs and Collis (1982). Their tool for studying student products is called the Structure of the Observed 
Learning Outcome, or SOLO. An apparent extension of that system, called the Conceptual/Integrative Coding System, was developed 
by Suedfeld, Tetlock, and Streufert (1992). These tools provided assistance in analyzing student products to determine whether 
students had, in fact, gone beyond mere reporting to develop synthetic insights. The research also provided additional support for 
Walvoord and Anderson's (1998) system of developing clearly defined rubrics for guiding and evaluating student work. 

A Common Deflnition for Assessment 

As described by the American Association for Higher Education, assessment is 

an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and 
public;settingappropriate criteriaand high standards for learningquality;systematicallygathering, analyzing, and interpreting 
evidence to determine how well performance matches those expectations and standards; and using the resulting information 
to document, explain, and improve performance (Angelo, 1995, p. 7). 

This definition suggests that faculty must understand and specify what they expect from their students. Further, it suggests that they 
describe in their syllabi and assignments what is expected in a manner that undergraduate students can effectively interpret and 
address. Finally, the definition assumes that faculty can assess their own and their students' work to make judgments about 
improvements in future iterations of the course. 

The AAHE definition suggests the need to align the parts of the curriculum as Ewell (1997) described it in order for the maximum 
learning experience to be achieved. Consequently, in my research (Canavan, 2001) I applied these concepts to describe how 
professors designed courses; articulated their course objectives to the students; delivered the learning experience through course 
activities; established the expectations for student performance; and, finally, understood and interpreted students' experience 
of the course. 




One ofthe most interesting challenges in my research was to find a tool that would provide a relatively consistent means of defining 
and assessing “synthesis” in the student products submitted to meet the requirements their professors established for them. 
Synthesis is the one common learning goal established by the faculty ofthe University for the Tier ill courses 1 used for my research. 
My search for useful tools led me to the work of Biggs and Collis (1982), Biggs (1999b). and Suedfeld, Tetlock, and Streufert (1992). 
The taxonomies or coding systems these researchers had developed to work with a variety of student products gave me tools to 
approach the assessment of synthesis in student papers. 
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More importantly, I have concluded through my research that a more valuable use of these tools is to assist faculty in determining 
appropriate learning experiences for students in their courses and to further align their course goals and the students’ experiences 
of them. By using SOLO or the Conceptual/Integrative (C/I) Coding System in the design of courses, faculty could more effectively plan 
class activities focusing on an appropriately high level of intellectual activity considered to be the standard of collegiate work. 

In the sections following I outline the components of the curriculum in the courses contained in this research as Ewell (1997) describes 
them: (1) designed, (2) expectational, (3) delivered, and (4) experienced. Faculty course proposals, syllabi, criteria for evaluation, and 
student products were all part of the data reviewed. Finally, I suggest that SOLO (Biggs & Collis, 1982) or the C/I Codingtool (Suedfeld, 
Tetlock, & Streufert, 1992) could provide tools useful to faculty in the course development process. 



Designing Courses and Setting Expectations 

The first components of Ewell’s (1997) alignment model are the designed curriculum and the expectational curriculum. How does a 
department or a professor design the learning experience in a course? What learning objectives (expectations) are established for 
students? The courses I used in my research have a shared learning objective of synthesis. Because this is the focus established by 
the faculty of the university, my research focused on curricular alignment based on this learning objective. The primary question was: 
Do students in these interdisciplinary courses experience the primary learningoutcome of synthesis in theircourses and demonstrate 
it in their class papers? 

Faculty members participate in a faculty development workshop to prepare courses for the program. The University Curriculum Council 
must approve their proposals for the courses they design, so a substantial effort goes into the formulation of courses for the program. 
The result of these efforts is a broad range of courses drawing upon the disciplinary interests of a substantial number of faculty 
members. Each course must involve perspectives from two or more disciplines and teach students to synthesize their learning from 
these disciplines. 



Course Syllabi 

Studentswhosign upforthecourses described in my research do not see thecourse proposals that constitutethe primary descriptions 
of the designed courses. What students see are catalog descriptions and syllabi. In addition, they receive clarifying information and 
specific course assignments throughout the term. The course syllabus, however, is the primary tool for establishing course goals and 
criteria for student success that derive from the designed course outlined by the professor in the proposal submitted to the University 
Curriculum Council. 

Interestingly, the course syllabi that professors provided for the study were surprisingly void of student learning outcomes directed 
at synthesis. In fact, only two of the ten courses in the study listed explicit learning outcomes at all, although several stipulated course 
goals. On the other hand, two of the syllabi listed criteria for assessing student papers that approach the grading rubrics 
recommended by Walvoord and Anderson (1998). These descriptions indicated to students that a good deal more was expected from 
them than merely parroting library research. 

Writing learning outcomes to elicit the demonstration of synthesis is not an easy task. The Bloom definition of synthesis that is the 
old standard demonstrates the problem: 

The putting together of elements and parts so as to form a whole. This is a process of working with elements, parts, etc., and 
combining them in such a way as to constitute a pattern or structure not clearly there before (Bloom, Engelhart, Furst, Hill. & 

Krathwohl. 1956, p. 162). 

Faculty descriptions of expectations for learning in courses designed to “teach” synthesis have to “operationalize” it in terms more 
clearly understood by students. The activities designed to get at synthesis have to demand more than analysis of the literature and 
“reassembly” of those perspectives so that the student reveals “something not clearly there before.” 



Assignments for Synthesis 




The assignments required in a course furtherdemonstrate the design a faculty member has developed to achieve his or her goals. The 
assignments typically required bythecourses are substantial essaysor group projects. Additionally, a number ofthe courses included 
journaling as a part ofthe course requirements. The specifics of these assignments were not detailed on most of the course syllabi. 
Several professors gave students additional handouts describing the major assignments later in the course (Canavan, 2001). 
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The selection of papers, journaling, and group projects have been found to support the development of synthesis and/or critical 
thinking (Baxter-Magolda, 1992; Farmer & Napieralski, 1997; Wiggins. 1993). However. I found that many of the assignment 
descriptions for these courses asked students to “analyze” or “compare and contrast.” These verbs appear in Bloom’s taxonomy at 
a level below synthesis and do not indicate to students thatthey are expected to do more than collect appropriate references and report 
on them. 

□ Delivered Curriculum 

Ewell’s (1997) third component of curriculum is the delivered curriculum: what actually goes on in the classroom. This part of the 
curriculum should be. and often is. the most stimulating and enrichingcomponent. In the delivery of additional information and 
the exchanges between peers and instructor, the substance of the course evolves. All of the professors involved in the study 
spoke very fondly of the courses they taught and the learning that they believed went on in those exchanges. Indeed. Baxter- 
Magolda’s work (1992) reported the educational value students place on the interaction among peers at the upper levels of the 
undergraduate curriculum. Students at these upper levels see that each of them has as much to contribute to their learning as 
the professor. Baxter-Magolda’s research was supported in the interviews I conducted with students from the Tier III courses. 

□ The Experienced Curriculum 

Many participating students reported that their class had provided a meaningful experience, and that they had benefited from 
this experience that drew them away from their major departments at an interesting time in their undergraduate careers. As 
seniorsthey had been immersed in majorcoursesand lookingahead tograduation in a short period oftime. In theirTier III courses 
theywere required to mingle with students from otherdisciplinaryareasand to tap into those resources as well as to explore what 

was often substantially a new area of study for them. 

□ Assessing Student Products for Synthesis 

The final task was to assess the student products for achievement of synthesis. My review of individual course syllabi had 
provided no cleardescriptionor rubric. soldecided toapply SOLO (Biggs&Collis.i982)and C/I CodingSystem(SuedfeldTetlock. 
&Streufert. 1992) to the analysis (see the appendixes). SOLO was developed to assistteachers in designing curriculum by helping 
them establish appropriate levels ofchallenge for students. The C/I CodingSystem was developed by psychologists as a tool for 
assessing the complexity of student thinking when responding to specific writing prompts. Needless to say. neither system is 
foolproof, especially when applied to full-length papers of the type submitted forthese interdisciplinary courses. Nevertheless, 
they did provide a framework for evaluating the extent to which student products demonstrated synthesis. 

The examination of forty-three papers from the ten courses provided a range of student ability and varietyof approaches to the 
assignments, as one would certainly expect. By their very nature, the assignments were not consistent in the demand for 
synthesis. Few of the assignments really indicated that a high level of synthesis was expected. Indeed, several of the ten 
professors still used terms like “analyze” and “compare and contrast” in their assignment descriptions. It was not surprisingthat 
twenty-seven of the forty-three papers exhibited little more than combining of findings from library resources. It occurred to me 
that the tools I had found would likely serve faculty as they go about the process of developing learning objectives and course 
assignments. 



Using Solo and the C/I Coding System to Guide Curricular Alignment 

Biggs (1999b) also speaks to the need to align the curriculum. In this work, and in an article he published for the AAHE (1999a). he 
speaks to the need to establish high goals for student learning. Presumably, when faculty use the SOLO taxonomy or the C/I 
descriptors, they can write essay prompts or assignment descriptions to elicit higher order thinking skills. Both of these taxonomies 
or coding systems indicate that, as cognitive development advances, the ability to envision multiple possibilities and to be able to 
explain and compare alternatives should be expected. Asking students to develop plans for solving problems based on their research 
would be a better alternative than simply asking them to describe and compare solutions that have been attempted (library research). 
Additionally, working within groups on such problems provides a real-life component of give and take as a group works toward a 
solution acceptable for presentation to the class. 



Conclusion 



The examination of the Tier III courses indicated that few faculty members in the study had “aligned” courses. They could do more to 

describe learningobjectives concisely and clearly inorderto prompt studentdevelopmentofsynthesis. The useoftheSOLOtaxonomy 

or the Conceptual/Integrative Coding descriptors could provide useful tools to guide the development of their learning objectives. A 
logical extension ofthat work would be thedevelopment of grading rubrics fortheir assignments such as those described byWalvoord 
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and Anderson {1998). Following the assessment of student work, faculty could return to their course designs to implement change 
based on the information obtained. SOLO and the C/I descriptors would provide benchmarks against which to compare their own 
learning objectives and rubrics so that a consistently high level of intellectual activity is demanded in the courses they teach. 
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Appendix A 

SOLO Summaty (Biggs & Collis, 1982, p. 25) 



Table 2.1 

Base Stage of Cognitive Development and Response Description* 



Developmental 
base stage with 
minimal age 


SOLO 

Description 


1 

Capacity 


2 

Relating operation 


3 

Consistency 
and closure 


4 

Response Structure 
Cue Response 


Formal 
Operations 
(16+ years) 


Extended 

Abstract 


Maximal: cue + 
relevant data * 
interrelations 
+ hypotheses 


Deduction and 
induction. 

Can generalize to 
situations not 
experienced 


Inconsistencies resolved. 
No felt need to give closed 
decisions-conclusions 
held open, or qualified 
to allow logically 
possible alternatives. 

(Ri. R2, or R3) 


X 

X ^ 


Concrete 
Generalization 
(13-15 years) 


Relational 


High: cue + 
relevant data 
interrelations 


Induction. Can 
generalize within 
given or experienced 
context using related 
aspects 


No inconsistency within 
the given system, 
but since closure is unique 
so inconsistencies may 
occur when he goes 
outside the system 


X 

X 

0 


Middle 
Concrete 
(io-i2 years) 


Multistructural 


Medium: cue 
isolated relevant data 


Can “generalize" 
only in terms of 
a few limited and 
Independent aspects 


Although has a feeling 
for consistency can be 
inconsistent because 
closes too soon on basis of 
isolated fixations on data, 
and so can come to 
different conclusions with 
same data 


X 

X 

X 

• 

0 

0 

0 


Early Concrete 
(7-10 years) 


Unistructural 


Low: cue -t- 
one relevant datum 


Can “generalize" 
only in terms of 
one aspect 


No felt need for 
consistency, thus closes 
too quickly; jumps to 
conclusions on one 
aspect, and so can be 
very inconsistent 


X 

X 

X 

• 

▲ 

• 

• 

0 

0 

0 


Pre-operational 
(4-6 years) 


Prestructural 


Minimal: cue and 
response confused 


Denial, tautology, 
transduction. 
Bound to specifics 


No felt need for 
consistency. 
Closes without even 
seeing the problem 


X 

• 

• 

0 

0 

0 




‘Kinds of data used: X = irrelevant or inappropriate; • = related and given in display: O = related and hypothetical, not given. 
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Appendix B 

Integrative Complexity Coding System Summary 

(Suedfeld, Tetlock, a Streufert, 1992) 

SCORE OF 1: No conceptual differentiation or integration: simple, one-dimensional rules for interpreting events or making choices. 

1. Com pa rtmenta lization 

a. Categorical rejection of perspectives: avoidance of values trade-offs 

b. Setting up and knocking down a “straw man” 

c. Inclusion-exclusion rules 

2. Dominance of a single evaluative rule 

a. Lack of response differentiation 

b. Use of lists 

3. Conflict avoidance 

4. Prescriptive generalizations 

5. Temporal sequencing 

Flags: absolutely, all, always, certainly, constantly, convinced, definitely, entirely, forever, impossible, indisputable, irrefut- 
able, irreversible, never, solely, surely, unconditionally, undoubtedly, unquestionably... 

SCORE OF 2: Emergent differentiation (still categorical in nature). 

1. Conditional acceptance of other perspectives or dimensions 

2. Conditional statements 

3. Conditions of a hypothetical outcome 

4. Exceptions to the rule 

5. Emergent recognition of alternate perspectives or dimensions 

6. Increased tolerance of alternate perspectives or dimensions 

Flags: but, nevertheless, while, however, though, probably, almost, usually... 

SCORE OF 3: Clear differentiation but no evidence of integration. Recognition of alternative perspectives with different dimensions 
as relevant and legitimate. 

1. Multiple alternatives: must be clearly two or more categories or rule structures to warrant a score of 3 

a. Multiple perspectives 

b. Multiple dimensions 

c. Multiple perspectives and multiple dimensions 

2. Alternatives and conditions for application: complex conditional reasoning 

3. Probability statements: independent causes or determinants 

4. Temporal perspectives: sensitivity to impact of experience on perception 

5. Increased tolerance for ambiguity: parallel, contradictory perspectives 

Flags: all those for score of 2 plus: alternatively, either-or, on the other hand, meanwhile... 

SCORE OF 4: Emergingintegration: multiple perspectives-dimensions existand they could interact. At leastone ofthe following must 
be found. 

1. Withholding judgment (further information needed) 

2. Tension between alternatives 

3. Integration expressed probabilistically (it is likely, it seems possible, they will probably...) 

4. Integration expressed as a super ordinate statement: broad statement which encompasses the multiple dimensions 
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SCORE OF 5; Explicit integration: multiple interactive dimensions. 

1. Mutual influence and interdependence: dynamic relationships 

2. Negotiation: trade-off reasoning 

3. Causal attributions 

4. Synthesis: generation of a novel product 

Flags: interplay, interaction, interdependency, mutuality, compromise, equilibrium, balancing, trade-off...ln general, however, 
content flags are not useful in determining a score at this level. 

SCORE OF 6: High-level interaction working across multiple levels of schemata. At least one indicator must be present. 

1. Comparison of outcomes 

2. Systematic analysis 

3. Hypothesis testing: set values on variables in the system and predict how the system would react to a new variable 

SCORE OF 7: Presence of an overarching viewpoint: Explanation of organizing principles (temporal, causal, theoretical). Discussion 

of ways in which levels of the problem/concept interact (demonstrating the validity of the overarching view). 

1. Hierarchical integration: principles/concepts which offer an explanation for a particular event, problem, or theory 

2. Comparison of outcomes 

3. Systematic analysis: specific complex interactions within a complex system, using an overarching global view (a rippling 
effect) 

4. Complex trade-offs among conflicting goals: cost-benefit analyses, for example 
Coding notes: Assign the highest justifiable score. 
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